Description: Candidates should be able to troubleshoot networking issues on client hosts.
Key Knowledge Areas:
Manually and automatically configure network interfaces and routing tables to include adding, starting, stopping, restarting, deleting or reconfiguring network interfaces
Change, view, or configure the routing table and correct an improperly set default route manually
Debug problems associated with the network configuration
Terms and Utilities:
Till now we have learned about fundamentals of internet protocols and we have get familiar with some of network configuration files and utilities. The truth is that some times things doesn't work as we expected and need troubleshooting. In this section we try to show some steps to resolve the problem, additionally some new commands will be introduced.
The first command we have learned is ifconfig . Some times there might be an inactive interface which doesn't appear in results:
[email protected]:~# ifconfiglo Link encap:Local Loopbackinet addr:127.0.0.1 Mask:255.0.0.0inet6 addr: ::1/128 Scope:HostUP LOOPBACK RUNNING MTU:65536 Metric:1RX packets:6 errors:0 dropped:0 overruns:0 frame:0TX packets:6 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:318 (318.0 B) TX bytes:318 (318.0 B)
use -a with ifconfig or ip command instead:
[email protected]:~# ifconfig -aens33 Link encap:Ethernet HWaddr 00:0c:29:e2:1b:3eBROADCAST MULTICAST MTU:1500 Metric:1RX packets:44 errors:0 dropped:0 overruns:0 frame:0TX packets:113 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:5169 (5.1 KB) TX bytes:12521 (12.5 KB)lo Link encap:Local Loopbackinet addr:127.0.0.1 Mask:255.0.0.0inet6 addr: ::1/128 Scope:HostUP LOOPBACK RUNNING MTU:65536 Metric:1RX packets:6 errors:0 dropped:0 overruns:0 frame:0TX packets:6 errors:0 dropped:0 overruns:0 carrier:0collisions:0 txqueuelen:1000RX bytes:318 (318.0 B) TX bytes:318 (318.0 B)[email protected]:~# ip a s1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever2: ens33: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000link/ether 00:0c:29:e2:1b:3e brd ff:ff:ff:ff:ff:ff
Bring up the interface with
ifup ens33 or
ifconfig ens33 up , and next check for your ip address.
[email protected]:~# ifup ens33
You can check your ip address either from GUI or trough config files.If you are on Automatic ip assignment use
dhclientto release and renew your ip address.
ping is our best friend when we are troubleshooting network problems.
Check whether you can ping another computer in the same network?
[email protected]:~# ping 172.16.43.127 -c 2PING 172.16.43.127 (172.16.43.127) 56(84) bytes of data.64 bytes from 172.16.43.127: icmp_seq=1 ttl=64 time=0.748 ms64 bytes from 172.16.43.127: icmp_seq=2 ttl=64 time=0.755 ms--- 172.16.43.127 ping statistics ---2 packets transmitted, 2 received, 0% packet loss, time 1001msrtt min/avg/max/mdev = 0.748/0.751/0.755/0.027 ms
from a simple ping command we can determine whether the target is up and running or not. Also there might be a firewall in your network, which filters out ICMP packets, check for host firewall first and then hardware firewall if there are any.
[email protected]:~# ping 172.16.43.126 -c 2PING 172.16.43.126 (172.16.43.126) 56(84) bytes of data.From 172.16.43.135 icmp_seq=1 Destination Host UnreachableFrom 172.16.43.135 icmp_seq=2 Destination Host Unreachable--- 172.16.43.126 ping statistics ---2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1007mspipe 2
Some times you might ping a wrong ip address or the server might have two interfaces or two ip adresses.
Regular ping command only works with IPv4 address. Use ping6 command to send ICMPv6 ECHO_REQUEST packets to network hosts from a host or gateway.
If you cant reach any network except computers you are in the same subnet with, you should doubt about you gateway.
[email protected]:~# routeKernel IP routing tableDestination Gateway Genmask Flags Metric Ref Use Ifacedefault 172.16.43.2 0.0.0.0 UG 0 0 0 ens33link-local * 255.255.0.0 U 1000 0 0 ens33172.16.43.0 * 255.255.255.0 U 0 0 0 ens33
we can also use netstat -rn command to see current gateway:
[email protected]:~# netstat -rnKernel IP routing tableDestination Gateway Genmask Flags MSS Window irtt Iface0.0.0.0 172.16.43.2 0.0.0.0 UG 0 0 0 ens33169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ens33172.16.43.0 0.0.0.0 255.255.255.0 U 0 0 0 ens33
If there were no default gate way, you should use
route add default gw x.x.x.x to add a default gateway. Next check the gate way, and make sure packets are going out from the right interface:
[email protected]:~# ping -c 3 172.16.43.2PING 172.16.43.2 (172.16.43.2) 56(84) bytes of data.64 bytes from 172.16.43.2: icmp_seq=1 ttl=128 time=0.434 ms64 bytes from 172.16.43.2: icmp_seq=2 ttl=128 time=0.379 ms64 bytes from 172.16.43.2: icmp_seq=3 ttl=128 time=0.166 ms--- 172.16.43.2 ping statistics ---3 packets transmitted, 3 received, 0% packet loss, time 2025msrtt min/avg/max/mdev = 0.166/0.326/0.434/0.116 ms
Every thing seems okey, but you cant reach specific ip address in another building. Hmm there might be routing problems in physical routers! how to check that?
The traceroute command maps the journey that a packet of information undertakes from its source to its destination. This tool uses ICMP messages, but unlike ping, identifies every router in the path. traceroute is useful when troubleshooting network problems because it can help you to localize problems in network connectivity (you might need to install it
apt install traceroute):
[email protected]:~# traceroute google.comtraceroute to google.com (22.214.171.124), 30 hops max, 60 byte packets1 172.16.43.2 (172.16.43.2) 0.272 ms 0.329 ms 0.172 ms2 172.16.130.1 (172.16.130.1) 0.969 ms 1.183 ms 1.128 ms3 192.168.1.66 (192.168.1.66) 0.448 ms 0.574 ms 0.655 ms4 192.168.66.41 (192.168.66.41) 5.123 ms 4.924 ms 7.027 ms5 * * *6 * * *7 10.10.53.93 (10.10.53.93) 14.334 ms 14.304 ms 14.256 ms8 10.201.147.214 (10.201.147.214) 19.629 ms 19.580 ms 21.650 ms9 10.21.21.22 (10.21.21.22) 19.445 ms 16.203 ms 16.131 ms10 10.21.21.22 (10.21.21.22) 13.911 ms 13.793 ms 13.723 ms11 126.96.36.199 (188.8.131.52) 48.527 ms 48.476 ms 48.416 ms12 184.108.40.206 (220.127.116.11) 48.361 ms 48.285 ms 47.198 ms13 18.104.22.168 (22.214.171.124) 49.444 ms 49.409 ms 126.96.36.199 (188.8.131.52) 49.677 ms14 184.108.40.206 (220.127.116.11) 52.925 ms 46.305 ms 45.917 ms15 arn02s05-in-f142.1e100.net (18.104.22.168) 45.783 ms 45.708 ms 4
use -i to Specifies the interface through which traceroute should send packets.
traceroute with option
-6 supports ipv6, instead we can use traceroute6 command.
Tracepath traces a path to a designated network address, reporting on the "time to live" or TTL lag and maximum transmission units (MTU) along the way. This command can be run by any user other with access to the command line prompt.
[email protected]:~# tracepath google.com1?: [LOCALHOST] pmtu 15001: 172.16.43.2 0.148ms1: 172.16.43.2 0.139ms2: 172.16.130.1 4.944ms asymm 13: 192.168.1.66 1.224ms asymm 14: 192.168.66.41 8.198ms asymm 15: 192.168.198.169 5.567ms asymm 16: 192.168.0.254 5.923ms asymm 17: 10.10.53.93 10.018ms asymm 18: 10.201.147.214 11.534ms asymm 19: 10.21.21.22 15.286ms asymm 110: 10.21.21.22 11.588ms asymm 111: 22.214.171.124 43.501ms asymm 112: no reply...30: no replyToo many hops: pmtu 1500Resume: pmtu 1500
Traceroute is essentially the same as Tracepath except that by default, it will only give the TTL value.
tracepath6 is good replacement for traceroute6.
Dig stands for (Domain Information Groper) is a network administration command-line tool for querying Domain Name System (DNS) name servers. It is useful for verifying and troubleshooting DNS problems and also to perform DNS lookups and displays the answers that are returned from the name server that were queried.
By default, dig sends the DNS query to name servers listed in the resolver(/etc/resolv.conf) unless it is asked to query a specific name server.
[email protected]:~# dig aol.com; <<>> DiG 9.10.3-P4-Ubuntu <<>> aol.com;; global options: +cmd;; Got answer:;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51870;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 1;; OPT PSEUDOSECTION:; EDNS: version: 0, flags:; udp: 512;; QUESTION SECTION:;aol.com. IN A;; ANSWER SECTION:aol.com. 587 IN A 126.96.36.199aol.com. 587 IN A 188.8.131.52aol.com. 587 IN A 184.108.40.206aol.com. 587 IN A 220.127.116.11aol.com. 587 IN A 18.104.22.168;; Query time: 46 msec;; SERVER: 22.214.171.124#53(126.96.36.199);; WHEN: Mon Feb 24 16:12:46 +0330 2020;; MSG SIZE rcvd: 116
The dig command output has several sections sections , to have just Answer section use +short switch :
[email protected]:~# dig aol.com +short188.8.131.52184.108.40.206220.127.116.1118.104.22.16822.214.171.124
to query specific Name server use
[email protected]:~# dig aol.com @126.96.36.199 +short188.8.131.52184.108.40.206220.127.116.1118.104.22.16822.214.171.124
netstat (network statistics) is a command-line tool that displays network connections (both incoming and outgoing), routing tables, number of network interface and even network protocol statistics.
By default, netstat displays a list of open sockets .( A socket is one end-point of a two-way communication link between two programs running on the network.) as an example X11:
[email protected]:~# netstat | grep X11 | head -n3unix 3 [ ] STREAM CONNECTED 34942 @/tmp/.X11-unix/X0unix 3 [ ] STREAM CONNECTED 33525 @/tmp/.X11-unix/X0unix 3 [ ] STREAM CONNECTED 32753 @/tmp/.X11-unix/X0
So we usually use a combination of switches with netstat :
netstat command example
Listing all the LISTENING Ports of TCP and UDP connections
all LISTENING ports, but shows numerical addresses
Listing TCP Ports connections
Listing UDP Ports connections
Listing all LISTENING(TCP&UDP) Connections
Showing Statistics by Protocol(TCP&UDP&...)
Displaying Service name with PID
Displaying Kernel IP routing
use netstat in conjunction with grep to get a better results.
nc (or netcat) utility is used for just about anything under the sun involving TCP or UDP. It can open TCP connections, send UDP packets, listen on arbitrary TCP and UDP ports, do port scanning, and deal with both IPv4 and IPv6. Unlike telnet, nc scripts nicely, and separates error messages onto standard error instead of sending them to standard output, as telnet does with some.
[email protected]:~# netcat -l 8888
The -l parameter means that netcat is in listen (server) mode, and 8888 is the port it listens to; netcat will create a socket server and wait for connections on port 8888 . The terminal will remain on hold for a client to connect to the open server with netcat. We can verify that a host service listens on port 8888.We need to open a new terminal to the host station and run the command:
[email protected]:~# netstat -na | grep 8888tcp 0 0 0.0.0.0:8888 0.0.0.0:* LISTEN