Image
Capture packets in Kubernetes with this open source tool
Troubleshoot complex network and application issues with ksniff, a kubectl plugin that captures packets in Kubernetes pods.
Networking is a fundamental sysadmin skill, but it is often overlooked. Many sysadmins find networking topics challenging, and it can be difficult to progress from core network skills to advanced troubleshooting capabilities without regular practice. This may be why networking is a popular topic on Enable Sysadmin.
Kubernetes and its associated extensions, such as service meshes, introduce additional network complexity that an administrator must be prepared to tackle. Basic network tools, such as ping and traceroute, can be helpful during the initial troubleshooting stage. However, I've consistently found that viewing the packets traversing the wire is the best way to troubleshoot and understand complex protocol and application-level issues.
This article introduces ksniff, an excellent open source tool that I've been using. Ksniff is a plugin for kubectl that allows you to capture packets in your Kubernetes pods. I'll show you how to capture, filter, and save packets for later analysis. Let's get started!
Disclaimer: The techniques explained in this article should only be used in development environments and with a complete understanding of what you are doing. The official ksniff documentation does not yet recommend its use in production, and you should be aware that it uploads a precompiled tcpdump binary to your running pods.
Install ksniff and related tools
Ksniff is a plugin for kubectl, and you must install it before you can start using it. The official installation instructions for ksniff recommend using the Krew plugin manager, which is my preferred installation method. You can find installation instructions for Krew on the project's website. Once you have Krew installed, you can install ksniff with a single command:
$ kubectl krew install sniff
Updated the local copy of plugin index.
Installing plugin: sniff
Installed plugin: sniff
\
| Use this plugin:
| kubectl sniff
| Documentation:
| https://github.com/eldadru/ksniff
| Caveats:
| \
| | This plugin needs the following programs:
| | * wireshark (optional, used for live capture)
| /
/
WARNING: You installed plugin "sniff" from the krew-index plugin repository.
These plugins are not audited for security by the Krew maintainers.
Run them at your own risk.
If you want to follow along and view captured packets in a graphical user interface (GUI) or terminal, you should also install Wireshark and tshark.
[ You might be wondering: Red Hat OpenShift and Kubernetes ... what's the difference? ]
Set up a workload to test
The best way to get started with ksniff is to perform a packet capture on a familiar workload, such as a basic web server. I recommend a simple Nginx pod, which you can run with a single command. These examples deploy a simple workload on a single-node K0s installation running on my workstation:
$ kubectl run --image=nginx nginx
pod/nginx created
$ kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx 1/1 Running 0 4s
Once the pod is running, you can expose it as a NodePort service. While most admins generally avoid NodePort services in production, they provide a convenient way to test a service quickly:
$ kubectl expose pod nginx --port 80 --type=NodePort
service/nginx exposed
$ kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 2d5h
nginx NodePort 10.99.139.251 <none> 80:32209/TCP 2s
Finally, you can confirm that the service is operable with a simple curl to the NodePort service:
$ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
acritelli-k8s Ready control-plane 3d5h v1.23.3+k0s 10.10.0.207 <none> Ubuntu 20.04.4 LTS 5.14.0-1029-oem containerd://1.5.9
$ curl 10.10.0.207:32209
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
[ Free online course: Red Hat Enterprise Linux technical overview. ]
Capture packets
Once you have installed ksniff and have a pod running, it's time to capture some network traffic. Running kubectl sniff $POD_NAME will begin a packet capture, launch Wireshark, and send the packet capture directly to Wireshark:
$ kubectl sniff nginx
INFO[0000] using tcpdump path at: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'nginx'
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] sniffing on pod: 'nginx' [namespace: 'default', container: 'nginx', filter: '', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx'
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx', pod: 'nginx', namespace: 'default'
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''
INFO[0000] file found: ''
INFO[0000] file was already found on remote pod
INFO[0000] tcpdump uploaded successfully
INFO[0000] spawning wireshark!
INFO[0000] start sniffing on remote container
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx', pod: 'nginx', namespace: 'default'
Image
[ Get this free eBook: Managing your Kubernetes clusters for dummies. ]
If you prefer to stay entirely at the command-line interface (for example, if you are on a remote server), then you can also send ksniff's output to the tshark terminal program:
$ kubectl sniff nginx -o - | tshark -r -
INFO[0000] using tcpdump path at: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'nginx'
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] sniffing on pod: 'nginx' [namespace: 'default', container: 'nginx', filter: '', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx'
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx', pod: 'nginx', namespace: 'default'
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''
INFO[0000] file found: ''
INFO[0000] file was already found on remote pod
INFO[0000] tcpdump uploaded successfully
INFO[0000] output file option specified, storing output in: '-'
INFO[0000] start sniffing on remote container
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx', pod: 'nginx', namespace: 'default'
1 0.000000 10.244.0.1 → 10.244.0.18 TCP 76 29681 → 80 [SYN] Seq=0 Win=65495 Len=0 MSS=65495 SACK_PERM=1 TSval=420487628 TSecr=0 WS=128
2 0.000023 10.244.0.18 → 10.244.0.1 TCP 76 80 → 29681 [SYN, ACK] Seq=0 Ack=1 Win=64260 Len=0 MSS=1440 SACK_PERM=1 TSval=3356168433 TSecr=420487628 WS=128
3 0.000040 10.244.0.1 → 10.244.0.18 TCP 68 29681 → 80 [ACK] Seq=1 Ack=1 Win=65536 Len=0 TSval=420487628 TSecr=3356168433
4 0.000069 10.244.0.1 → 10.244.0.18 HTTP 149 GET / HTTP/1.1
5 0.000072 10.244.0.18 → 10.244.0.1 TCP 68 80 → 29681 [ACK] Seq=1 Ack=82 Win=64256 Len=0 TSval=3356168433 TSecr=420487628
6 0.000160 10.244.0.18 → 10.244.0.1 TCP 306 HTTP/1.1 200 OK [TCP segment of a reassembled PDU]
7 0.000178 10.244.0.1 → 10.244.0.18 TCP 68 29681 → 80 [ACK] Seq=82 Ack=239 Win=65408 Len=0 TSval=420487628 TSecr=3356168433
8 0.000192 10.244.0.18 → 10.244.0.1 HTTP 683 HTTP/1.1 200 OK (text/html)
9 0.000198 10.244.0.1 → 10.244.0.18 TCP 68 29681 → 80 [ACK] Seq=82 Ack=854 Win=64896 Len=0 TSval=420487628 TSecr=3356168433
10 0.000296 10.244.0.1 → 10.244.0.18 TCP 68 29681 → 80 [FIN, ACK] Seq=82 Ack=854 Win=65536 Len=0 TSval=420487628 TSecr=3356168433
11 0.000326 10.244.0.18 → 10.244.0.1 TCP 68 80 → 29681 [FIN, ACK] Seq=854 Ack=83 Win=64256 Len=0 TSval=3356168433 TSecr=420487628
12 0.000354 10.244.0.1 → 10.244.0.18 TCP 68 29681 → 80 [ACK] Seq=83 Ack=855 Win=65536 Len=0 TSval=420487628 TSecr=3356168433
^C⏎
Real-time analysis is useful, but saving a packet capture for later analysis is also common. This is especially helpful if you capture packets on a machine such as a jump host that does not have access to Wireshark or tshark. You can write packets to a file by specifying an output to ksniff. You can open the saved file in a protocol analyzer, such as Wireshark:
$ k sniff nginx -o /tmp/nginx_capture.pcap
INFO[0000] using tcpdump path at: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'nginx'
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] sniffing on pod: 'nginx' [namespace: 'default', container: 'nginx', filter: '', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/home/acritelli/.krew/store/sniff/v1.6.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'nginx'
INFO[0000] executing command: '[/bin/sh -c test -f /tmp/static-tcpdump]' on container: 'nginx', pod: 'nginx', namespace: 'default'
INFO[0000] command: '[/bin/sh -c test -f /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''
INFO[0000] file found: ''
INFO[0000] file was already found on remote pod
INFO[0000] tcpdump uploaded successfully
INFO[0000] output file option specified, storing output in: '/tmp/nginx_capture.pcap'
INFO[0000] start sniffing on remote container
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'nginx', pod: 'nginx', namespace: 'default'
^C⏎
$ file /tmp/nginx_capture.pcap
/tmp/nginx_capture.pcap: pcap capture file, microsecond ts (little-endian) - version 2.4 (Linux cooked v1, capture length 262144)
Packet captures can become very messy, especially for pods that run complex or highly utilized workloads. The ability to apply tcpdump filters is critical to simplifying complex packet captures, and ksniff supports this out of the box:
$ kubectl sniff nginx -f "tcp port 80"
Wrap up
This article shows you how to leverage ksniff to capture packets inside of running Kubernetes pods. Packet captures provide a powerful method to observe and troubleshoot complex network and application issues, and their applicability is even more relevant as environments become increasingly complicated.
Performing packet captures is also a great way to sharpen your networking skills and develop a deeper understanding of complex environments, so I recommend you leverage a tool like ksniff at every available opportunity. Good luck, and happy networking!
Image
Redundancy with automated failover is good. But making the wrong decisions can make a high-availability solution worse than no redundancy at all.
Image
Use Nmap, tcpdump, and Wireshark to discover why your enterprise switch and clients aren't communicating IP configurations correctly.
Image
Networking is one of a sysadmin’s most important duties, so make sure you have the essentials covered.
Anthony Critelli
Anthony Critelli is a Linux systems engineer with interests in automation, containerization, tracing, and performance. He started his professional career as a network engineer and eventually made the switch to the Linux systems side of IT. He holds a B.S. and an M.S. More about me
Try Red Hat Enterprise Linux
Download it at no charge from the Red Hat Developer program.
Related Content
Image
Now that your environment has been set up, deploy a sample application on an OpenShift Local cluster.
Image
Install Red Hat OpenShift Local on your own machine to test your work before deployment.
Image
Get greater control over TCP port checking with a DIY, customizable approach using Python and Scapy.