Network troubleshooting with packet captures

You're in the middle of troubleshooting some weird network or service issue, things just aren't right, the ports are open, the service is listening, but something is still off. Beyond verifying the basics, connectivity, ports, and running services, sometimes you just need to see what's going through the network. Stick around, and we'll talk about a few ways you can snoop on what's running across the wire, or up and down your application stack.

Why capture packets?

Networks and services can be complex. One service reaches out to another to get some bit of information, then that one reaches out to something else, and so on. Everything has to mesh. Name lookups and other dependent services need to function correctly. There's also latency. Wouldn't it be great to figure out where a request is getting bogged down? You can get some of this information in other ways, but actually seeing what's going down the wire can be immensely useful. On top of that, if you're in a secure, or segmented network, you may run into blocked ports. Port scanning can help a lot here.

Simply knowing that traffic can pass isn't always enough. What you'd like to see is exactly what traffic is passing, and what it is saying. Fortunately, there is a way to snoop on that traffic and see what's inside of each and every packet. It's called packet sniffing, or packet capture.

Capturing DHCP packets

First, let's determine a test service and then look at what a packet capture from that service can tell us. We're going to use DHCP as a good test because it's easy to follow, and readily available to me as I write this.

I'll spare you the background on how DHCP works and summarize a bit. When a new host that is configured to use DHCP is brought online, it first asks for an IP via a broadcast (a request sent out that all active devices with an IP on that subnet will see). If there is a DHCP server available, that server responds. How it responds depends on whether it thinks it's allowed to assign an IP, and if it has any IP addresses available in its pool.

You can test this yourself with a basic virtual machine (VM) setup. I have one virtual guest running on my laptop, using libvirt. That guest is configured to get an IP address directly from the NAT network configured by libvirt, which also provides a DHCP server. I ran a packet capture using tcpdump on my host machine while I powered on my guest machine. We'll talk more about how to use the tcpdump command in a moment.

I am displaying the results with Wireshark, another tool we'll talk about later.

Image

The first thing we see in the packet capture is a system with no IP address asking for an IP, and then our DHCP server responding. The response, if you dig a little deeper, contains all the information that the requesting system needs to configure its local IP address.

Image

The Option: (xx) fields are various extra bits of information the DHCP server is configured to provide.

So, you can imagine how useful this tool is when troubleshooting a DHCP issue. You can not only see if your DHCP server responded but what it responded with! If you never saw the initial DHCPREQUEST, it could tell you that your client isn't connected to the proper network, or has a network problem. If you saw the DHCPACK, but some of the info looks wrong, you could try to pinpoint a configuration issue.

Using tcpdump

So how do you perform this wizardry? It's not terribly difficult. All you need to get started is a simple tool called tcpdump. You can get it from yum on a RHEL system. Once installed, you can check out its very extensive man page for more information, but the basics are simple. You need root privileges to gather this sort of information. You need to know what interface to expect traffic on (you can listen on all interfaces, but that gets very chatty), and you'll want to decide how to output the data. In this case, I'm writing to a file. I used the following command for my example:

[nlager@nlager ~]$ sudo tcpdump -i virbr1 -w ./virt-interface1.pcap

After you've performed whatever task you're trying to troubleshoot, hit Ctrl-C to end the capture. You should get some output telling you how many packets tcpdump captured:

[nlager@nlager ~]$ sudo tcpdump -i virbr1 -w ./virt-interface1.pcap
tcpdump: listening on virbr1, link-type EN10MB (Ethernet), capture size 262144 bytes
^C46 packets captured
54 packets received by filter
0 packets dropped by kernel

[nlager@nlager ~]$

And you should have an output file:

nlager@nlager ~]$ ls -l virt-interface1.pcap
-rw-r--r--. 1 tcpdump tcpdump 5293 Mar  5 12:55 virt-interface1.pcap

That file is binary, though, so you can't just cat it and see what's in there. You can read it with tcpdump, but a more helpful tool is something like Wireshark (again, available via yum). You'd install Wireshark, then load in the pcap file. Personally, I run all of my servers without a GUI, so I keep Wireshark on my workstation. After you complete your packet capture, copy it to your workstation and import the pcap file into Wireshark. Browse to File -> Open, and then select your pcap file. Wireshark can also do live packet captures, but tcpdump is much more convenient when you're working with a server because it may not have a GUI. The packet capture includes a lot more information than I showed above. The full window looks like this:

Image

Clicking on an item in the list of packets in the top pane gives you a decoded view of the data in the middle pane. The bottom pane shows you the raw hex of the packet.

Packet captures are also very useful to support engineers when they're trying to help you troubleshoot odd network behavior. A packet capture can tell you how long it took for a remote resource to respond, what it responded with, and whether that data looks sane or not.

Security concerns

Be aware, though, that packet captures can contain anything that the system you are running it on can see on the network, not just what's available inside its own TCP stack. If I am using some unencrypted protocols, like telnet, HTTP, FTP, DNS, or many others, tcpdump can happily snoop on that data and save it to your file. Wireshark can also decode SSL encrypted traffic, and re-assemble it if you can provide it with a certificate. It can even re-construct VoIP calls! If a bad guy gets into one of your systems and finds tcpdump, they'll be thankful that you've left such a valuable tool at their disposal. So, it would be a good idea to remove tcpdump immediately after solving whatever problem you're attempting to troubleshoot.

Certain network-level safeguards can be put in place to help prevent a system from capturing all of the traffic on a given network. So if you're worried about curious users sniffing their co-worker's banking details, fear not, modern network switches disable promiscuous mode to stop such bad actors.

Wrapping up

I hope you've found this to be a good introduction to network troubleshooting with packet captures. Hopefully, you can see how valuable, and how dangerous, packet captures can be.

[ Want to do more with your network? Download a free ebook on network automation with Ansible. ]