You're in the middle of troubleshooting some weird network or service issue, things just aren't right, the ports are open, the service is listening, but something is still off. Beyond verifying the basics, connectivity, ports, and running services, sometimes you just need to see what's going through the network. Stick around, and we'll talk about a few ways you can snoop on what's running across the wire, or up and down your application stack.
Why capture packets?
Networks and services can be complex. One service reaches out to another to get some bit of information, then that one reaches out to something else, and so on. Everything has to mesh. Name lookups and other dependent services need to function correctly. There's also latency. Wouldn't it be great to figure out where a request is getting bogged down? You can get some of this information in other ways, but actually seeing what's going down the wire can be immensely useful. On top of that, if you're in a secure, or segmented network, you may run into blocked ports. Port scanning can help a lot here.
Simply knowing that traffic can pass isn't always enough. What you'd like to see is exactly what traffic is passing, and what it is saying. Fortunately, there is a way to snoop on that traffic and see what's inside of each and every packet. It's called packet sniffing, or packet capture.
Capturing DHCP packets
First, let's determine a test service and then look at what a packet capture from that service can tell us. We're going to use
DHCP as a good test because it's easy to follow, and readily available to me as I write this.
I'll spare you the background on how
DHCP works and summarize a bit. When a new host that is configured to use
DHCP is brought online, it first asks for an IP via a broadcast (a request sent out that all active devices with an IP on that subnet will see). If there is a
DHCP server available, that server responds. How it responds depends on whether it thinks it's allowed to assign an IP, and if it has any IP addresses available in its pool.
You can test this yourself with a basic virtual machine (VM) setup. I have one virtual guest running on my laptop, using
libvirt. That guest is configured to get an IP address directly from the NAT network configured by
libvirt, which also provides a
DHCP server. I ran a packet capture using
tcpdump on my host machine while I powered on my guest machine. We'll talk more about how to use the
tcpdump command in a moment.
I am displaying the results with
Wireshark, another tool we'll talk about later.
The first thing we see in the packet capture is a system with no IP address asking for an IP, and then our
DHCP server responding. The response, if you dig a little deeper, contains all the information that the requesting system needs to configure its local IP address.
Option: (xx) fields are various extra bits of information the
DHCP server is configured to provide.
So, you can imagine how useful this tool is when troubleshooting a
DHCP issue. You can not only see if your
DHCP server responded but what it responded with! If you never saw the initial
DHCPREQUEST, it could tell you that your client isn't connected to the proper network, or has a network problem. If you saw the
DHCPACK, but some of the info looks wrong, you could try to pinpoint a configuration issue.
So how do you perform this wizardry? It's not terribly difficult. All you need to get started is a simple tool called
tcpdump. You can get it from
yum on a RHEL system. Once installed, you can check out its very extensive
man page for more information, but the basics are simple. You need root privileges to gather this sort of information. You need to know what interface to expect traffic on (you can listen on all interfaces, but that gets very chatty), and you'll want to decide how to output the data. In this case, I'm writing to a file. I used the following command for my example:
[nlager@nlager ~]$ sudo tcpdump -i virbr1 -w ./virt-interface1.pcap
After you've performed whatever task you're trying to troubleshoot, hit Ctrl-C to end the capture. You should get some output telling you how many packets
[nlager@nlager ~]$ sudo tcpdump -i virbr1 -w ./virt-interface1.pcap tcpdump: listening on virbr1, link-type EN10MB (Ethernet), capture size 262144 bytes ^C46 packets captured 54 packets received by filter 0 packets dropped by kernel [nlager@nlager ~]$
And you should have an output file:
nlager@nlager ~]$ ls -l virt-interface1.pcap -rw-r--r--. 1 tcpdump tcpdump 5293 Mar 5 12:55 virt-interface1.pcap
That file is binary, though, so you can't just
cat it and see what's in there. You can read it with
tcpdump, but a more helpful tool is something like
Wireshark (again, available via
yum). You'd install
Wireshark, then load in the
pcap file. Personally, I run all of my servers without a GUI, so I keep
Wireshark on my workstation. After you complete your packet capture, copy it to your workstation and import the
pcap file into
Wireshark. Browse to File -> Open, and then select your
Wireshark can also do live packet captures, but
tcpdump is much more convenient when you're working with a server because it may not have a GUI. The packet capture includes a lot more information than I showed above. The full window looks like this:
Clicking on an item in the list of packets in the top pane gives you a decoded view of the data in the middle pane. The bottom pane shows you the raw hex of the packet.
Packet captures are also very useful to support engineers when they're trying to help you troubleshoot odd network behavior. A packet capture can tell you how long it took for a remote resource to respond, what it responded with, and whether that data looks sane or not.
Be aware, though, that packet captures can contain anything that the system you are running it on can see on the network, not just what's available inside its own TCP stack. If I am using some unencrypted protocols, like telnet, HTTP, FTP, DNS, or many others,
tcpdump can happily snoop on that data and save it to your file.
Wireshark can also decode SSL encrypted traffic, and re-assemble it if you can provide it with a certificate. It can even re-construct VoIP calls! If a bad guy gets into one of your systems and finds
tcpdump, they'll be thankful that you've left such a valuable tool at their disposal. So, it would be a good idea to remove
tcpdump immediately after solving whatever problem you're attempting to troubleshoot.
Certain network-level safeguards can be put in place to help prevent a system from capturing all of the traffic on a given network. So if you're worried about curious users sniffing their co-worker's banking details, fear not, modern network switches disable promiscuous mode to stop such bad actors.
I hope you've found this to be a good introduction to network troubleshooting with packet captures. Hopefully, you can see how valuable, and how dangerous, packet captures can be.
[ Want to do more with your network? Download a free ebook on network automation with Ansible. ]