How to use the lsof command to troubleshoot Linux
The Linux lsof command does more than list open files; you can also use it to diagnose potential bottlenecks.
One of the things I appreciate most about Linux is the chance to explore my operating system. Nothing is off limits in Linux. That's a huge opportunity—a very comfortable benefit—for anyone using Linux daily, whether you're a sysadmin, developer, network admin, or hobbyist eager to learn more.
It has real-world ramifications, too, because sometimes you really do need to probe into the inner workings of a system to understand a problem. Every sysadmin knows the feeling of getting a vague message like "The server is slow" or "This application isn't responding." That lucky sysadmin gets to perform a "treasure hunt" to find the problem. However, Linux provides all the tools you need.
[Cheat sheet: Old Linux commands and their modern replacements ]
List open files
lsof command is an acronym for "list open files," but its potential isn't limited to just that role. It's commonly said that in Linux, everything is a file. In many ways, that's true, so a utility that lists open files is actually pretty useful. The
lsof utility is a robust interface for the information inside the
/proc virtual filesystem.
For example, suppose you need to know what process is using a particular directory:
$ lsof /run COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME systemd 1 root 31u FIFO 0,24 0t0 19314 /run/initctl systemd 1 root 38u FIFO 0,24 0t0 19331 /run/dmeventd-server systemd 1 root 39u FIFO 0,24 0t0 19332 /run/dmeventd-client systemd-j 713 root mem REG 0,24 8388608 12871 /run/log/journal/aaff550ef54147ddb4d1776d86b04055/system.journal NetworkMa 881 root 20w FIFO 0,24 0t0 24896 /run/systemd/inhibit/1.ref rhsmcertd 906 root 3wW REG 0,24 0 22479 /run/lock/subsys/rhsmcertd crond 910 root 3uW REG 0,24 4 24002 /run/crond.pid rsyslogd 1224 root mem REG 0,24 8388608 12871 /run/log/journal/aaff550ef54147ddb4d1776d86b04055/system.journal rsyslogd 1224 root 10r REG 0,24 8388608 12871 /run/log/journal/aaff550ef54147ddb4d1776d86b04055/system.journal sshd 1651 root 11w FIFO 0,24 0t0 27390 /run/systemd/sessions/1.ref sshd 1666 admin 11w FIFO 0,24 0t0 27390 /run/systemd/sessions/1.ref
Or maybe you need the exact opposite. Perhaps you need to know what files a particular process has open. This example looks at
$ lsof -p 890 CMD PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 890 root cwd DIR 253,0 224 128 / sshd 890 root rtd DIR 253,0 224 128 / sshd 890 root txt REG 253,0 886344 646480 /usr/sbin/sshd sshd 890 root mem REG 253,0 6940392 34389337 /var/lib/sss/mc/group sshd 890 root mem REG 253,0 9253600 34389336 /var/lib/sss/mc/passwd sshd 890 root mem REG 253,0 46312 570898 ...libnss_sss.so sshd 890 root mem REG 253,0 144392 1847 ...libgpg-error.so [...] sshd 890 root 0r CHR 1,3 0t0 1027 /dev/null sshd 890 root 1u unix 0xff... 0t0 22395 type=STREAM sshd 890 root 2u unix 0xff... 0t0 22395 type=STREAM sshd 890 root 4u unix 0xff... 0t0 24654 type=STREAM sshd 890 root 5u IPv4 24658 0t0 TCP *:ssh (LISTEN) sshd 890 root 6r REG 253,0 6940392 34389337 /var/lib/sss/mc/group sshd 890 root 7u IPv6 24666 0t0 TCP *:ssh (LISTEN)
Here's an explanation of each column:
- COMMAND: The command name
- PID: Process ID (PID) of the process
- USER: Owner of the process
- FD: File descriptor definition
- TYPE: Type of file descriptor
- DEVICE: Device number or, in the case of a block device, character or other
- SIZE/OFF: Dimension of the file or offset (the suffix
0tis the offset)
- NODE: Node description of the local file; this could be the number of the local file, TCP, UDP, or STR (stream)
- NAME: The name of the mount point where the file resides
You can also discover what files a particular user has open:
$ lsof -u admin
This information is especially useful when you can't unmount an external device because a file on it is being actively accessed.
[ Get the guide to installing applications on Linux. ]
As I've said, everything on Linux is a file, so
lsof isn't limited to the local filesystem. You can also use it for network debugging.
For example, suppose you need to know what process uses a particular TCP port (like 22, for example):
$ lsof -i TCP:22 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME sshd 890 root 5u IPv4 24658 0t0 TCP *:ssh (LISTEN) sshd 890 root 7u IPv6 24666 0t0 TCP *:ssh (LISTEN)
You can also obtain a report on all network processes associated with a particular interface:
$ lsof -i TCP@127.0.0.1 COMMAND PID USER FD TYPE DEVICE SZ/OFF NODE NAME cupsd 4408 root 8u IPv4 32331 0t0 TCP ...ipp (LISTEN) qemu-s 13462 qemu 11u IPv4 98536 0t0 TCP ...rfb (LISTEN)
[ For more tips on Linux administration, read How to analyze a Linux process' memory map with pmap. ]
Despite its seemingly humble purpose, the
lsof tool is a powerful tool on a system that can treat almost anything as a file. Read the
lsof man page to get more ideas of how this simple utility can help you perform some complex tasks.
Use the pmap command to explore how a process is mapped in memory to monitor or troubleshoot memory usage.
Every sysadmin needs a good troubleshooting strategy, and you can't fix a problem if you cannot identify it. These are my favorite commands to quickly filter through the possibilities of a given problem.
There are a few commands that should always be in your sysadmin toolbox. Get to know these 7 essential networking commands.