Manipulating text at the command line with sed

In a previous article, I covered how to manipulate text with grep. Now, turn your attention to the sed (Stream Editor), which is best suited to be used in pipelines (data that comes from a pipe). The sed utility can be used to print the contents of a file, substitute a line (or multiple lines), and then save the file. In contrast to grep, sed can substitute a line or multiple lines in a file and perform an in-place update of that file.

The simplest sed invocation when substituting foo for bar is:

$ sed 's/foo/bar/' inputfile

Example: Remove comments

While grep can format output on screen, it is unable to modify a file in place. To do this, you’d need a file editor like ed . Since ed is not part of this article, use sed to achieve the same thing you did with grep in the previous article's first example. This time modify the /etc/fstab file in-place passing the -i flag to sed . Without the -i flag , you'd only see what would have been modified.

You are encouraged to always run sed without the -i flag, just to make sure that the result it produces is expected. The sed utility also offers the -i.bak flag, which creates a backup file before editing.

The final grep command for this example was:

$ grep -v '^#' /etc/fstab > ~/fstab_without_comment

With sed, you have:

# sed -i '/^#/d' /etc/fstab

/dev/mapper/VGCRYPTO-ROOT /                       ext4    defaults,x-systemd.device-timeout=0 1 1
UUID=e9de0f73-ddddd-4d45-a9ba-1ffffa /boot                   ext4    defaults             1 2
LABEL=SSD_SWAP		swap		swap		defaults	0 0

Example: Print only /etc/passwd users

In the grep example, you printed only usernames from the /etc/passwd file with the following:

$ grep -Eo '^[a-zA-Z_-]+' /etc/passwd

You can do the same using sed as follows:

$ sed 's/^\([a-zA-Z_-]\+\).*/\1/' /etc/passwd

In the above example, you group a match by parentheses (), and then print the matched group with \1 (back-reference), which dictates the first group. For a second group, you’d use \2, and so on.

Example: Replace all foo with bar

In sed, you can search for a pattern and then replace only the occurrence matching the pattern. To replace all occurrences in the file inputfile1 from foo to bar globally, run:

$ sed -i '/foo/bar/g' inputfile1

Example: Replace a single instance

Take the file inputfile2, which has the following contents:

hello world
second line should be replaced
this line should be replaced later

Say that you want to replace should with will, but only for the second line. This command breaks down as follows:

$ sed '/second/s/should/will/' inputfile2
          |    |    |     |
          |    |    |     with this pattern 
          |    |    this pattern
          |    substitute   
         Search for the pattern "second"

This output is sent to standard output rather than replacing the file's contents. The result looks like this:

$ sed '/second/s/should/will/' inputfile2
hello world
second line will be replaced
this line should be replaced later

The sed command is case-sensitive. The following won’t work when you try to replace World with there:

$ echo "Hello World" | sed 's/world/there/'
Hello World

GNU sed introduced a new flag, /I, which ignores the case and will perform the replacement with the same command:

$ echo "Hello World" | sed 's/world/there/I'
Hello there

Example: Print a range of lines and quit

With sed, you can also print lines and quit after your criteria are met. The following commands will print three lines and quit. This command:

$ sed -n '1,3p' /etc/passwd

is equivalent to:

$ sed '3q' /etc/passwd

The following would be wrong:

$ sed '1,3q' /etc/passwd # Wrong. You cannot quit three times 

Example: Comment out uncommented lines

Regular expressions can also be used with sed, as demonstrated earlier. For example, you have the following small script:

$ cat test_script 
#/usr/bin/env bash

this is the first comment
This is another comment
# this is a comment too

echo "This is not a comment and should be echoed"

You now have to skip the first line, starting with #!/bin/bash, and comment out the third and fourth lines, but not the fifth because that line is already commented.

In sed, you can use something like the following:

$ sed '3,6s/^[^#]/# &/g' test_script
#/usr/bin/env bash

# this is the first comment
# This is another comment
# this is a comment too

echo "This is not a comment and should be echoed"

In the above command, the following is performed:

• 3,6s defines a range, from line three down to line six.
• /^[^#]/ matches everything that is a character and does not start with a hash (#).
• /# &/g replaces a part, in this case it puts a # in front of the line dictated by the & sign.

Example: Remove all digits

Different applications generate data in different formats. With sed, you can keep only the data you can use. For example, you have the following file (inputfile3) in this format:

foo1234
bar99128
baz2842
qux12953
discard39120

Maybe a program generated the wrong format, or it concatenated the fields to one. What if you were only interested in keeping the alpha characters and wanted to discard the digits? How would you achieve this goal with sed?

The answer is probably easier than you think:

$ sed 's/\([a-z]*\).*/\1/' inputfile3
foo
bar
baz
qux
discard

Example: Change specific lines

Furthermore, sed can also handle ranges by pattern, which means you can specify a start and an end string and manipulate the range. For example:

$ cat inputfile4

hello world

start of the comment
another comment
end of a comment

dont comment this line
nor this line

The following sed command will comment lines starting with start and ending with end:

$ sed '/start/,/end/ s/^/# /' inputfile4 
hello world

# start of the comment
# another comment
# end of a comment

dont comment this line
nor this line

Get rid of the empty lines as well.

$ sed '/start/,/end/ s/^/# /;/^$/d' inputfile4

hello world
# start of the comment
# another comment
# end of a comment
dont comment this line
nor this line

There is much more to sed and its rich features. To be able to fully utilize sed’s abilities, please see its documentation page, which you can find here. Also, a great source of information on sed can be found here.

Wrap-up

As I covered previously, you'll use grep when you want to search for a pattern, either in a file or multiple directories recursively.  Use sed if you are receiving data from a pipeline, or want to manipulate data on the fly.

The sed command is scripted and it’s easy to learn to perform basic operations. All you need is practice, especially with regular expressions.