Common Troubleshooting Tools
The most fundamental troubleshooting technique for network devices is to merely
determine whether a device is reachable. Simply put, is the device up or is it down. To
this end, the Packet Internet Groper (PING) utility was developed to provide a simple
way to determine the reachability of a device.
Determining Reachability
PING is built upon the Internet Control Message Protocol (ICMP) and uses a system of
echo and echo-reply messages to indicate whether a host is reachable. When a source
host attempts to determine whether a destination host is reachable, it generates an ICMP
echo packet for the destination host. When the destination host receives the echo packet,
it responds with an echo-reply packet, allowing the source host to ascertain that the
destination is reachable. If for some reason the destination host does not respond, the
assumption is that it is unreachable. It is important to note that this is an assumption,
because there are any number of reasons a host might not respond to an echo packet even
though it is actually up and accessible. Therefore, it is important to understand that the
failure to receive a response to an echo message does not necessarily provide any
information regarding what, if anything, might be wrong. It simply means that the source
host did not receive a response to the echo message.
In the event that a host fails to respond to an echo request, two common messages may be
generated. The first is a "request timed out" response. This typically indicates that the
destination network was able to be reached but that the destination host did not respond
(which indicates a problem with the destination host itself). The other response is
"destination network unreachable." This message indicates that the destination network
was unable to be located and typically indicates a problem with the network
interconnectivity or a routing problem, not necessarily a problem with the destination
host.
As mentioned in Chapter 3, "TCP/IP for Firewalls," because of the nature of ICMP
traffic, it is a best practice to restrict ICMP through the firewall to prevent ICMP-based
attacks from being directed against hosts protected by the firewall. The obvious side
effect is that if ICMP is blocked, PING cannot be used through the firewall to
troubleshoot potential network problems. A commonly implemented workaround for this
is to allow certain types of ICMP traffic through the firewall. These workarounds include
allowing echo-reply, time-exceeded, and unreachable messages from untrusted hosts,
while allowing echo messages to untrusted hosts. This allows you to ping external hosts
and provides a means for the firewall to allow the three common echo message responses
to return. This can be done on the Cisco Secure PIX Firewall by applying the commands
in Example A-1 as a component of an access list.
Example A-1. ACL to Permit Only Certain ICMP Message Types
access-list 100 permit icmp any any echo-reply
access-list 100 permit icmp any any time-exceeded
access-list 100 permit icmp any any unreachable
As a best practice, your external firewall interface (and all corresponding IP addresses)
should not allow any other ICMP traffic. This will prevent someone from being able to
ping the firewall external IP address to determine whether it is accessible and will also
protect against malicious ICMP-based traffic such as a "ping of death."
Another aspect of reachability is to show how the device was reachable. In other words,
what path through the network was taken to the destination host? To answer this
question, the traceroute utility (in Windows, the command is tracert) was developed.
Traceroute functions by sending messages that are designed to exceed the time-to-live
value of the next routing hop. This causes the router to respond with a time-exceeded
message. Traceroute then increments the time-to-live value (starting at 1, then 2, and so
on) and generates a new packet for the destination host, allowing the source host to
determine each hop that is traversed on the way to the destination host.
As previously mentioned, PING does not necessarily provide any information regarding
where a failure occurred that prevented a host from being reachable. As a result,
traceroute is commonly implemented in conjunction with a failed PING test in an attempt
to determine where in the network between the source and destination hosts that a failure
might have occurred. For example, if you know that there are seven hops between the
source and destination, and the destination fails to respond to an echo message, if the
traceroute fails to respond after the fourth hop, you know that you should start looking at
the network between the fourth and fifth hops as the likely cause of the reachability
failure.
Unlike PING, traceroute typically uses UDP datagrams for the transport mechanism. An
exception to this is the implementation on Windows, which uses ICMP PING packets. In
both cases, however, the IP time-to-live value and the corresponding ICMP time-
exceeded responses are the key to how traceroute functions. For example, in Example A-
2 a traceroute is issued against the remote destination 10.21.120.43, showing the three
routers that are located between the source and destination hosts.
Example A-2. Traceroute to Remote Destination
C:\Documents and Settings\wnoonan>tracert 10.21.120.43
Tracing route to 10.21.120.43 over a maximum of 30 hops
1 31 ms 31 ms 31 ms 10.1.1.1
2 31 ms 32 ms 32 ms 10.2.1.2
3 45 ms 33 ms 32 ms 10.2.10.19
4 32 ms 45 ms 31 ms 10.21.120.43
Trace complete.
Determining Which Physical Addresses Are Known
A fundamental aspect of network communications is the need for two hosts to be able to
physically identify and communicate with each other. This is handled in TCP/IP by the
Address Resolution Protocol (ARP). ARP builds and maintains a table of MAC address
to IP address associations, allowing a host to be able to physically identify local hosts and
address them accordingly. If the host has the wrong physical address, however, the data
will not be successfully delivered. This mis-addressing is similar to having the wrong
street address on a postal letter.
You can display a list of MAC addresses that a host knows about by running the arp
command. For Windows hosts, you run arp -a to show the contents of the ARP cache, as
displayed in Example A-3.
Example A-3. Displaying the ARP Cache on Windows
C:\Documents and Settings\wnoonan>arp -a
Interface: 192.168.1.114 --- 0x5
Internet Address Physical Address Type
192.168.1.97 00-0c-ce-e5-56-71 dynamic
192.168.1.100 00-01-02-29-f4-28 dynamic
For many Linux-based hosts, just run arp, as shown in Example A-4.
Example A-4. Displaying the ARP Cache on Red Hat Linux
[wnoonan@keoland wnoonan]$ arp
Address HWtype HWaddress Flags Mask Iface
192.168.1.114 ether 00:0B:DB:21:C7:81 C eth0
192.168.1.101 ether 00:11:11:AF:E4:85 C eth0
192.168.1.97 ether 00:0C:CE:E5:56:71 C eth0
Windows-Specific Tools
In a GUI world, the command-line utility IPCONFIG is the best and most accurate way
to determine the IP address that a Windows computer is running. The reason for this is
simple: dynamically assigned IP addresses and IP address information is not shown in the
GUI network properties of an interface.
In addition to simply providing information regarding the IP address of the host,
IPCONFIG can also provide Domain Name Server (DNS) information, local MAC
addresses, name resolution server information, and Dynamic Host Configuration Protocol
(DHCP) server, and DHCP lease information. You can obtain this information by running
the command ipconfig /all, as shown in Example A-5.
Example A-5. Windows IP Configuration
C:\Documents and Settings\wnoonan>ipconfig /all
Windows IP Configuration
Host Name . . . . . . . . . . . . : host01
Primary Dns Suffix . . . . . . . : mydom.local
Node Type . . . . . . . . . . . . : Hybrid
IP Routing Enabled. . . . . . . . : No
WINS Proxy Enabled. . . . . . . . : No
DNS Suffix Search List. . . . . . : mydom.local
cisco.com
Ethernet adapter Local Area Connection:
Connection-specific DNS Suffix . : mydom.local
Description . . . . . . . . . . . : 3Com 3C920 Integrated Fast Ethernet
Controller (3C905C-TX Compatible) #2
Physical Address. . . . . . . . . : 00-0B-DB-21-C7-81
Dhcp Enabled. . . . . . . . . . . : Yes
Autoconfiguration Enabled . . . . : Yes
IP Address. . . . . . . . . . . . : 192.168.1.14
Subnet Mask . . . . . . . . . . . : 255.255.255.0
Default Gateway . . . . . . . . . : 192.168.1.1
DHCP Server . . . . . . . . . . . : 192.168.1.10
DNS Servers . . . . . . . . . . . : 192.168.1.10
192.168.1.11
Primary WINS Server . . . . . . . : 192.168.1.10
Secondary WINS Server . . . . . . : 192.168.1.11
Lease Obtained. . . . . . . . . . : Monday, September 19, 2005 19:12:34
Lease Expires . . . . . . . . . . : Monday, October 03, 2005 19:12:34
Linux- and UNIX-Specific Tools
Similar to the Windows IPCONFIG command, IFCONFIG is used to display IP
addressing configuration information on UNIX- and Linux-based hosts. In addition to
merely displaying the IP addressing configuration, you can use IFCONFIG to actually
configure the appropriate IP settings for an interface. Example A-6 shows this IP
information.
Example A-6. Red Hat Linux IP Configuration
[wnoonan@keoland wnoonan]$ ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:D0:09:DC:B4:2B
inet addr:192.168.1.118 Bcast:192.168.1.127 Mask:255.255.255.224
inet6 addr: fe80::2d0:9ff:fedc:b42b/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:2443 errors:0 dropped:0 overruns:0 frame:0
TX packets:201 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:224572 (219.3 Kb) TX bytes:30513 (29.7 Kb)
Interrupt:11 Base address:0xe000
Packet-Analysis Tools
As mentioned in Chapter 3, "TCP/IP for Firewalls," TCP/IP is the "language" that most
network-connected hosts use to communicate with each other. Packet-analysis tools
enable you to view the raw transmitted data, providing an incredibly valuable
troubleshooting technique. Through the use of tools such as Ethereal, Microsoft Network
Monitor, and TCPDump, you can observe all aspects of network communications
between hosts, allowing you to detect and identify network-based problems and
communications errors. Packet-analysis tools also enable you to answer the question of