Working with pipes on the Linux command line
I'm sorry to inform you, but the command line didn't die off with the dinosaurs, nor did it disappear with the dodo or the carrier pigeon. The Linux command line is alive and well, and still going strong. It is an efficient way of quickly gathering and processing information, creating new scripts, and configuring systems.
One of the most powerful shell operators is the pipe (|
). The pipe takes output from one command and uses it as input for another. And, you're not limited to a single piped command—you can stack them as many times as you like, or until you run out of output or file descriptors.
One of the main purposes of piping is filtering. You use piping to filter the contents of a large file—to find a particular string or word, for example. This purpose is why the most popular use for pipes involves the commands grep
and sort
. But, you're not limited to those cases. You can pipe the output to any command that accepts stream input. Let's look at a theoretical example as an illustration of how this process works:
$ cmd1 | cmd2
Both cmd1
and cmd2
are command line utilities that output their results to the screen (stdout
). When you pipe one command's output to another, however, the information from cmd1
doesn't produce output to the screen. The pipe redirects that output as input to cmd2
.
Note: Don't confuse pipe (
|
) redirection with file redirection (>
) and (<
). (An example of file redirection:cmd1 > file
orcmd1 < file
.) File redirection either sends output to a file instead of the screen, or pulls input from a file.
Let's look at some real-world examples of how piping works.
Checking on NICs
Let's say that you need to know if one of your network interface cards (NICs) has an IP address beginning with 192:
$ ifconfig | grep 192
inet 192.168.1.96 netmask 255.255.255.0 broadcast 192.168.1.255
You can also find out which live NICs you have on a system with a simple pipe to grep
:
$ ifconfig | grep UP
enp0s3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
You could also grep for "RUNNING" or "RUN" to display the same information.
Examining permissions
Maybe you want to find out how many directories under /etc
are writeable by root:
$ sudo ls -lR | grep drwx
The results are too long to list here, but as you can see from your displayed list, there are a lot of them. You still need to find out how many there are, and a visual count would take a long time. An easy option is to pipe the results of your ls
command to the wc
(word count) command:
$ sudo ls -lR | grep drwx | wc -l
285
The -l
switch displays the number of lines. Your count might be different. This listing is from a fresh "RHEL 8 server no GUI" install.
[Want to try out Red Hat Enterprise Linux? Download it now for free.]
Counting files
You don't have to use grep
all the time. For example, you can list the number of files in the /etc
directory with this:
$ ls | wc -l
229
Again, your results might look different, but you know something is wrong if the command returns a small number of files.
Identifying processes
You can also perform complex tasks using pipes. To list the process IDs (PIDs) for all systemd
-related processes:
$ ps -ef | grep systemd | awk '{ print $2 }'
1
471
500
608
631
7449
7494
17242
The awk
command's $2
output isolates the second (PID) column from the ps
command. Note that the last entry, PID 17242
, is actually the PID for the grep
command displaying this information, as you can see from the full listing results here:
khess 17242 7505 0 09:40 pts/0 00:00:00 grep --color=auto systemd
To remove this entry from your results, use the pipe operator again:
$ ps -ef | grep systemd | awk '{ print $2 }' | grep -v grep
The -v
switch tells the grep
command to invert or ignore lines that contain the string that follows—in this case any line containing the word "grep."
Sorting results
Another popular use of the pipe operator is to sort your results by piping to the sort
command. Say that you want to sort the list of names in contacts.txt
. First, let's look at the contents as they are in the file before sorting:
$ cat contacts.txt
Bob Jones
Leslie Smith
Dana David
Susan Gee
Leonard Schmidt
Linda Gray
Terry Jones
Colin Doe
Jenny Case
Now, sort the list:
$ cat contacts.txt | sort
Bob Jones
Colin Doe
Dana David
Jenny Case
Leonard Schmidt
Leslie Smith
Linda Gray
Susan Gee
Terry Jones
You can reverse the sort order with the -r
switch:
$ cat contacts.txt | sort -r
Terry Jones
Susan Gee
Linda Gray
Leslie Smith
Leonard Schmidt
Jenny Case
Dana David
Colin Doe
Bob Jones
Was the output for either of these what you expected? By default, the sort
command performs a dictionary sort on the first word or column, which is why Bob Jones
and Terry Jones
are not listed one after the other.
You can sort by the second column with the -k
switch, specifying the column you want to sort on, which in our list is column two:
$ cat contacts.txt | sort -k2
Jenny Case
Dana David
Colin Doe
Susan Gee
Linda Gray
Bob Jones
Terry Jones
Leonard Schmidt
Leslie Smith
If you have a long contact list and you think that this list contains duplicate entries, you can pipe your list to sort
and then uniq
to remove those duplicates:
$ cat contacts.txt | sort | uniq
This command only displays unique entries, but it doesn't save the results. To sort the file, filter unique entries, and then save the new file, use this:
$ cat contacts.txt | sort -k2 | uniq > contact_list.txt
Remember that the pipe operator and the file redirect operators do different things.
Wrapping up
Now that you've had a taste of the command line's power, do you think you can handle the responsibility? Command line utilities have more flexibility than their graphical counterparts do. And for some tasks, there are no graphical equivalents.
The best way to learn command line behavior is to experiment. For example, the sort
command didn't do what you thought it should until you explored further. Don't forget to use the power of your manual (man
) pages to find those "hidden" secret switches and options that turn command line frustration into command line success.
Ken Hess
Ken has used Red Hat Linux since 1996 and has written ebooks, whitepapers, actual books, thousands of exam review questions, and hundreds of articles on open source and other topics. Ken also has 20+ years of experience as an enterprise sysadmin with Unix, Linux, Windows, and Virtualization. More about me