改天翻译给大家，英语好的可以自行翻译
这里面缺少G.3. mtr G.5. tcpdump G.6. tcpflow G.7. tcpreplay
Appendix G. Diagnostic Tools
Table of Contents
Now that we have covered most of the basic tools for management of routes, IP addresses, and a few Ethernet tools, we come to a set of tools which are used primarily to help you figure out what is wrong in your network, where a route is broken, or even, simply, whether a host is reachable.
Some of these tools are available on other platforms, but may have different command line switches or may use different packet signatures than those described here. The concepts in many cases, transfer, but, of course, the command line options may be different.
We are going to start with one of the first networking tools that many people learn, ping and we'll move along to the common traceroute, which maps out a route from one host to another, mtr, which represents traceroute-type information in a richer format, netstat, for examining sockets (and routes) in use, and finally, the indispensable tcpdump, which reports on all traffic passing through a device.
By learning both how and when to use these tools, but even more importantly, how to read their output, you can perform a tremendous amount of reconnaisance on your own network and frequently quickly isolate problems and identify error conditions. These tools are some of the core tools of any linux administrator who is responsible for an IP network.
G.1. ping
ping is one of the oldest IP utilities around. Simply put, ping asks another host if it is alive, and records the round-trip time between the request and the reply.
In this section, we'll look at several examples of the use of ping to test reachability, send a specified number of packets, suppress all but summary output, stress the network, record the route a packet takes, set the TTL, specify ToS, and specify the source IP.
The ping utility has a simple and elegant design. When run, it will craft a packet bound for the specified destination, send the packet, and record the time it took that packet to reach its destination. The generated packet is an ICMP packet known as an echo-request. If the destination host receives the packet, it should generate an echo-reply. The success or failure of this very simple operation can provide some insight into the state of a network or a series of networks.
In most cases, the ICMP echo-request packets and echo-reply packets, upon which ping's functionality relies, are allowed through routers and firewalls, however with the advent of trojans and distributed denial of service tools which transmit information within ICMP packets, some networks and network administrators block ICMP at their borders. For an example of such a trojan, see this dissection of the trinoo distributed denial of service tool. As a result of these nefarious uses of echo-request and echo-reply packets, some cautious network administrators block all non-essential ICMP at their border routers. See Section 4.10, “ICMP and Routing” for a more complete discussion of ICMP.
Thus, we can no longer assume (as perhaps we once could) that simply because a host is not answering our ping request, this host is down. There may be a device which has been configured to filter out this traffic.
If a host is reachable and answering our echo-requests, then we may also wish to believe that the round-trip times recorded by ping are an accurate representation of network conditions. This can be misleading. Some routers are configured to give ICMP diagnostic messages the lowest priority of any IP packets travelling through them, in which case that router may contribute significantly to the round trip time of any echo-request packet passing through it.
With knowledge of these two potential roadblocks to the successful use of ping as a network diagnostic tool, we can begin to explore how ping is useful. In most internal networks, and many public networks, there are no filters to block our echo-request packets.
In its simplest form, ping is used interactively on the command line to test reachability of a remote host. Again, you'll see in all of the examples below the use of the -n switch to suppress DNS lookups. Since the proper functioning of DNS relies on a properly configured network, and ping is one of your tools for diagnosing network problems, it makes sense to suppress all name lookup until you have verified that the IP layer is functioning properly.
Let's see first if the host morgan can reach its default gateway. This example is similar to the test we performed in Example 1.2, “Testing reachability of a locally connected host with ping” from tristan.
On many systems, ping can be used by non-root users, but there are some options and features to ping which require the user to have administrative privilege or root-level access to the box. Therefore, all examples below will be run as the root user. Please be aware, that many diagnostics can be run without this high a level of privilege.
Example G.1. Using ping to test reachability
|
We have verified from morgan that its default gateway, branch-router is reachable. The first line of output tells us what the source and destination addresses (and names, if using DNS) are. Additionally, we learn the size of the data segment of the ping packet, 56 bytes, and the size of the entire outbound IP packet 84 bytes.
Each subsequent line of output before the summary is a record of the receipt of a reply from the destination (and what IP address sent the reply). Because ping needs to keep track of the number of bytes it has sent, and the round-trip time, each time you run ping, it creates a sequence number inside the data of the ping packet and reports the sequence number on any packets which return. By analyzing the timestamps on the returned packets, ping can determine the round trip time of the journey and reports this as the final field in each line of output.
At the end of the run, ping summarizes the number of replies, and performs some calculations on the round-trip times. As with much data collection, you need a large sample set of data to draw conclusions about your network. You can usually conclude that something is quite wrong if you cannot reach a remote host, but you should be cautious when concluding that your Ethernet card is bad simply because round-trip times to a destination on the LAN is high. It is more likely that there's another problem. Collecting ping data from a number of hosts to a number of destinations can help you determine if the problem is a localized to a single machine.
Frequently, you'll want to use ping in a script, or you'll want to specify that ping should only run for a few cycles. Fortunately, this is trivial (and I'll use the count option many times further below in this section). The -c restricts the number of packets which ping will send (or receive). It can be combined with some of the other options for a variety of diagnostic purposes.
Example G.2. Using ping to specify number of packets to send
|
In this example, we see a very regular 38 millisecond round trip time between morgan (192.168.98.82) and isolde (192.168.100.17). After sending 10 echo request packets and receiving the replies, ping summarizes the data for us and exits.
Occasionally, either in a script, or on the command line, you may not care about the output of each individual line. In this case, you can suppress everything except the summary data with the -q switch. In the following example, we are again testing reachability of isolde (192.168.100.17) though we only care about the summary output.
Example G.3. Using ping to specify number of packets to send
|
Here, we see only the output from ping as it begins to send packets to the destination, and the summary output when it has completed its run.
These are some simple examples of the use of ping to gather and present statistics on reachability of destination hosts, packet loss, and round trip times. Some other diagnostics information can be gathered with ping, too. Let's look at the use of ping to test reachability as aggressively as possible.
Occasionally, you'll want to stress the network to test how many packets you can squeeze through a link, and how gracefully performance on that link degrades. Fortunately, ping, when run with the -f switch can perform exactly this kind of test for you.
Example G.4. Using ping to stress a network
|
In this example, we have used the default packet size and sent 411 packets, receiving only 400 back from the remote host for a mere 2% packet loss. By increasing the packet size of the packet we are sending across the link we can get a sense for how quickly performance degrades on this link. If we use a much larger packet size (still smaller than Ethernet's 1500 byte frame), we see even more packet loss. We'll specify a packet size of 512 bytes with the -s option.
Example G.5. Using ping to stress a network with large packets
|
Flooding a low bandwidth link, like the ISDN link between morgan and masq-gw can be detrimental to other traffic on that link, so it is wise to use the -f with restraint. Although ping is a versatile tool for network diagnostics, it is not intended as a network performance measurement tool. For this sort of task, try netperf or collect some data with SNMP to analyze with MRTG.
As you can see, the use of ping floods is a good way to stress the network to which you are connected, and can be a good diagnostic tool. Be careful to stress the network for short periods of time if possible, or in a carefully controlled setting. Unless you want to alienate coworkers and anger your network administrator, you shouldn't start a ping flood and go home for the night.
The options we have outlined above are common options to ping, but now, let's look at some of the less common options. Occasionally, you may find yourself on a linux box without traceroute or mtr. Perhaps it's an embedded linux host, or a minimal installation with ping. There is an almost unknown option for recording the route a packet takes. By comparison to the more sophisticated tools for tracing network paths, ping with the record route option (-R) doesn't convey the information in as visually an appealing way, but it can get the job done.
Example G.6. Recording a network route with ping
|
As always, ping summarizes the output after it has completed its run, but let's examine the new section. By using the record route option, we are asking all routers along the way to include their IPs in the header. Although some routers may not observe this courtesy, many do. Unfortunately, there is only room to record 8 different hops (FIXME--verify this!), so the use of ping -R is mostly useful only in smaller networks.
The first IP we hit is our own IP on the way out our Ethernet interface, 192.168.98.82. Then it is a palindromic journey through the network stacks of each of the following hosts in order: branch-router, isdn-router, tristan, and back again in reverse order.
ping is even nice enough to report to us that a subsequent journey took the same route as the first packet. If you have a statically routed internal network, any subsequent packets should look exactly like the second packet. If dynamic routing is in use on your internal network, you may find that the routes change occasionally.
Now, frankly, I'm not sure of a practical use for the following option to ping, however, you can specify the TTL for an outbound echo requust packet. By setting the TTL you are specifying the maximum number of hops this packet will travel before it will be dropped. Conventionally, the TTL is set by the kernel to a reasonable number of hops (like 64). The -t provides us the capability to force the TTL for our echo requests. Now that we know it takes four hops to get to tristan from morgan we should be able to test whether setting the TTL makes any difference.
Example G.7. Setting the TTL on a ping packet
|
Clearly, we are able to reach tristan if we set the TTL on our echo requests to 4, but as soon as we drop the TTL to 3, we get a reply from the third hop (isdn-router), telling us that our packet was too old to be forwarded to its destination. If you are unclear on the rationale for TTL, I'd suggest reviewing some of the general IP documentation available in Section I.1.3, “General IP Networking Resources”.
Type of Service (ToS) is increasingly in use on backbones across the Internet which has brought with it Service Level Agreements (SLA). If you have an SLA with your provider, you may find the use of ping -Q to set the IP packet ToS flags will help you to determine if your provider is holding up their end of the bargain.
In Example G.8, “Setting ToS for a diagnostic ping” we'll set the ToS flag and verify with tcpdump that the ToS flag on the outbound packets have actually been set. Let's assume that we have an SLA with a backbone provider for our link between our German office (195.73.22.45) and our North American office (205.254.209.73). We'll send two test packets to the remote end, and observe the data on the wire.
Example G.8. Setting ToS for a diagnostic ping
|
Naturally, ping reports to us the round-trip times, the source and destination IPs, and that there was no packet loss. And our tcpdump output shows that the ToS flags were properly set on the packet. With all of this information, we can collect data about the reliability of the network between our two offices.
Occasionally, you'll find yourself on a heavily packet filtered host, or a host which employs conditional routing for packets with certain source addresses. Such packet filtering can prevent or conflict with the use of ping. Fortunately, ping allows the user to specify the source address of an outbound packet, thus allowing traversal of packet filters and conditional routing tables.
My classic example of a need for specifying source address on a ping is a VPN connected network. Let's assume masq-gw has a CIPEpeer in another city. Let's assume the internal IP on the peer is 192.168.70.254. If masq-gw sends a packet to the peer with a source address of 205.254.211.179, the peer might drop the inbound packet on a VPN interface from the public IP of the peer [59]. In this case, the peer should still accept traffic from masq-gw if the originating IP is inside the private network IP range.
In the Example G.9, “Specifying a source address for ping” we'll use ping to check reachability of the inside interface of the CIPE peer of masq-gw.
Example G.9. Specifying a source address for ping
|
By forcing the echo request packet to use the IP bound to one of our internal interfaces as the source address with the -I we are able to send traffic through the CIPE tunnel to the other side, and back.
As you can see, ping is a versatile tool in the network administrator's toolkit, and can be used for a wide range of tests beyond the simple reachability test. For a brief and humourous introduction to the program itself, see The Story of Ping.
Now that we have a good idea of the uses of the ping utility, let's move on to some other tools which can provide us other diagnostic data about our networks.
[59] If the admin controls both sides of the link, it is a matter of choice and preference whether traffic from the outside IP of the peer VPN endpoint should be allowed. I'll argue that traffic from the peer endpoint should not be allowed, but this is opinion only.
G.2. traceroute
traceroute is a utility for identifying the network path a packet will take to a destination. Like ping, it can be called a number of ways. traceroute takes advantage of a the TTL in an IP packet to determine hop by hop the reachability and addressing of routers between the traceroute host and the intended destination.
The tool traceroute is available on most Unix-like platforms and even under Windows as tracert. Here, we will only consider the common traceroute installed on linux systems.
The default packet type created by traceroute is a UDP packet. The first packet will be addressed to udp/33435 and each subsequent packet will be addressed to an incremented port number. This allows traceroute to keep track of which return ICMP packets correspond to which outbound packets.
Example G.10. Simple usage of traceroute
|
Note in Example G.10, “Simple usage of traceroute” that tcpdump conveniently reports the low TTL on the first packets. Packets transmitted from a router with a TTL of 1 will expire at the next router they hit. This is the concept and mechanism by which traceroute is able to detect the path by which packets arrive at their destination.
Each of the first three packets transmitted in the above example receive ICMP time exceeded replies from the upstream router (masq-gw). The second set of packets have their TTL set to 2, which is not reported by tcpdump. This allows these packets to reach the intended destination, tristan.
There is a liability of using UDP traceroute on the Internet. Many screening routers, firewalls, and even hosts will silently drop UDP packets, effectively destroying the usability of traceroute. On internal networks, or networks known to have no firewalls, conventional traceroute can continue to provide diagnostic value. In the case that the network is known to have a firewall, traceroute can use ICMP, and mtr is a good example of a network diagnostic tool which uses ICMP only.
G.4. netstat
The netstat utility summarizes a variety of characteristics of the networking stack. With netstat you can learn a number of important things. If no other type of data is requested it will report on the state of all active sockets. You can however request the routing table, masquerading table, network interface statistics, and network stack statistics [60].
One of the most common uses of the netstat utility is to determine the state of sockets on a machine. There are many questions that netstat can answer with the right set of options. Here's a list of some of the things different things we can learn.
-
which services are listening on which sockets
-
what process (and controlling PID) is listening on a given socket
-
whether data is waiting to be read on a socket
-
what connections are currently established to which sockets
By invoking netstat without any options, you are asking for a list of all currently open connections to and from the networking stack on the local machine. This means IP network connections, unix domain sockets, IPX sockets and Appletalk sockets among others. Naturally, we'll skip over the non-IP sockets since this is about IP networking with linux.
Assume the --inet switch in all cases below unless we are examining a particular higher layer protocol (e.g., TCP with the --tcp switch or UDP with --udp switch.
A convenient feature of netstat is its ability to differentiate between two different sorts of name lookup. Normally the -n specifies no name lookup, but this is ambiguous when there are hostnames, port names, and user names. Fortunately, netstat offers the following options to differentiate the different forms of lookup and suppress only the [un-]desired lookup.
-
--numeric-hosts -
--numeric-ports -
--numeric-users
The option -n (my favorite), suppress all hostname, port name and username lookup, and is a synonym for --numeric. I'll reiterate that hostnames and DNS in particular can be confusing, or worse, misleading when trying to diagnose or debug a networking related issue, so it is wise to suppress hostname lookups in these sorts of situations.
In Example G.11, “Displaying IP socket status with netstat” we will look at netstat's numeric output and then we'll invoke the same command but suppress the host lookups. Though the output is almost the same, a particular situation might call for one or the other invocation.
Example G.11. Displaying IP socket status with netstat
|
Each line represents a either the sending or receiving half of a connection. In the above output on morgan it appears that there are no connections other than TCP connections. If you are very familiar with TCP ports and the service associated with that port, then the first format will suffice in most cases. A possibly misleading aspect of the latter output is visible in the connections to and from localhost and the final line. netstat abbreviates the IP endpoint in order to reproduce the entire string retrieved from the port lookup (in /etc/services). Also interestingly, this line conveys to us (in the first output) that the kernel is waiting for the remote endpoint to acknowledge the 192 bytes which are still in the Send-Q buffer.
The first line describes a TCP connection to the IP locally hosted on morgan's Ethernet interface. The connection was initiated from an ephemeral port (40991) on tristan to a service running on port 22. The service normally running on this well-known port is sshd, so we can conclude that somebody on tristan has connected to the morgan's ssh server. The second line describes a TCP session open to port 993 on isolde, which probably means that the user on morgan has an open connection to an IMAP over SSL server.
The third through the sixth lines can be understood in pairs. By examining the source and destination IP and port pairs, we can see that two different TCP sessions have been established with the source and destination address of 127.0.0.1. For an administrator to publish services on localhost is not at all uncommon. This makes the service harder to abuse from the network. In this case, when we allow the service lookup, the port in question (6010) appears to be used to tunnel forwarded X applications over ssh.
The next line is the first TCP session in our output which is not in a state of ESTABLISHED. Refer to Table G.1, “Possible Session States in netstat output” for a full list of the possible values of the State field in the netstat output. The state SYN_SENT means that an application has made arequest for a TCP session, but has not yet received the return SYN+ACK packet.
The final line of our netstatoutput shows a connection in the TIME_WAIT state, which means that the TCP sessions have been terminated, but the kernel is waiting for any packets which may still be left on the network for this session. It is not at all abnormal for sockets to be in a TIME_WAIT state for a short period of time after a TCP session has ended.
If we needed to know exactly which application owned a particular network connection, we would use the -p | --program switch which gives us the PID and process name of the owner process. If we want to see the unix user and the PID and process we'll add the -e | --extend switch.
Example G.12. Displaying IP socket status details with netstat
|
There doesn't appear to be a large number of connections to and from the masq-gw host. The two sessions are initiated to the sshd running on port 22, and the process which owns each socket is a root process.
Table G.1. Possible Session States in netstat output
| State | Description |
|---|---|
| LISTEN | accepting connections |
| ESTABLISHED | connection up and passing data |
| SYN_SENT | TCP; session has been requested by us; waiting for reply from remote endpoint |
| SYN_RECV | TCP; session has been requested by a remote endpoint for a socket on which we were listening |
| LAST_ACK | TCP; our socket is closed; remote endpoint has also shut down; we are waiting for a final acknowledgement |
| CLOSE_WAIT | TCP; remote endpoint has shut down; the kernel is waiting for the application to close the socket |
| TIME_WAIT | TCP; socket is waiting after closing for any packets left on the network |
| CLOSED | socket is not being used (FIXME. What does mean?) |
| CLOSING | TCP; our socket is shut down; remote endpoint is shut down; not all data has been sent |
| FIN_WAIT1 | TCP; our socket has closed; we are in the process of tearing down the connection |
| FIN_WAIT2 | TCP; the connection has been closed; our socket is waiting for the remote endpoint to shut down |
One of the most common uses of netstat, especially in cross-platform environments is the reporting of the main routing table. On many platforms, netstat -rn is the preferred method of displaying routing information, although linux provides at least two alternatives to this: route and ip route show.
Example G.13. Displaying the main routing table with netstat
|
This output should look familiar. The routing cache itself may not be as familiar to most, but can also be displayed with netstat. The ouput below is exactly the same as the ouput from route -enC. Refer also to Example D.3, “Viewing the routing cache with route”.
Example G.14. Displaying the routing cache with netstat
|
Consult Section D.1.1, “Displaying the routing table with route” for more detail on reading and interpreting the data in this output. Because this is simply another way of reporting the routing table information, we'll skip over any detailed description.
netstat -i summarizes interface statistics in a terse format. This format
OK! This is strange. netstat -ie looks exactly like ifconfig output. That's weird!
For machines which perform masquerading, typically dual-homed packet-filtering firewalls like masq-gw a tool to list the current state of the masquerading table is convenient.
Each masqueraded connection can be described by a tuple of six pieces of data: the source IP and source port, the destination IP and destination port, and the (usually implicit) locally hosted IP and a local port.
FIXME; this command seems to fail on all of the iptables boxen, even if I'm using the -j MASQUERADE target. I can use it successfully on ipchains boxen. Anybody have any ideas or explanation here?
[60] Additionally, netstat can display multicast information with the --group switch. I have zero experience here. Anybody with experience want to write about this?
914
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
