In a previous article, I covered how to manipulate text with grep. Now, turn your attention to the sed (Stream Editor), which is best suited to be used in pipelines (data that comes from a pipe). The sed utility can be used to print the contents of a file, substitute a line (or multiple lines), and then save the file. In contrast to grep, sed can substitute a line or multiple lines in a file and perform an in-place update of that file.
The simplest sed invocation when substituting foo for bar is:
$ sed 's/foo/bar/' inputfile
Example: Remove comments
While grep can format output on screen, it is unable to modify a file in place. To do this, you’d need a file editor like ed . Since ed is not part of this article, use sed to achieve the same thing you did with grep in the previous article's first example. This time modify the /etc/fstab file in-place passing the -i flag to sed . Without the -i flag , you'd only see what would have been modified.
You are encouraged to always run sed without the -i flag, just to make sure that the result it produces is expected. The sed utility also offers the -i.bak flag, which creates a backup file before editing.
The final grep command for this example was:
$ grep -v '^#' /etc/fstab > ~/fstab_without_comment
With sed, you have:
# sed -i '/^#/d' /etc/fstab
/dev/mapper/VGCRYPTO-ROOT / ext4 defaults,x-systemd.device-timeout=0 1 1
UUID=e9de0f73-ddddd-4d45-a9ba-1ffffa /boot ext4 defaults 1 2
LABEL=SSD_SWAP swap swap defaults 0 0
Example: Print only /etc/passwd users
In the grep example, you printed only usernames from the /etc/passwd file with the following:
$ grep -Eo '^[a-zA-Z_-]+' /etc/passwd
You can do the same using sed as follows:
$ sed 's/^\([a-zA-Z_-]\+\).*/\1/' /etc/passwd
In the above example, you group a match by parentheses (), and then print the matched group with \1 (back-reference), which dictates the first group. For a second group, you’d use \2, and so on.
Example: Replace all foo with bar
In sed, you can search for a pattern and then replace only the occurrence matching the pattern. To replace all occurrences in the file inputfile1 from foo to bar globally, run:
$ sed -i '/foo/bar/g' inputfile1
Example: Replace a single instance
Take the file inputfile2, which has the following contents:
hello world
second line should be replaced
this line should be replaced later
Say that you want to replace should with will, but only for the second line. This command breaks down as follows:
$ sed '/second/s/should/will/' inputfile2
| | | |
| | | with this pattern
| | this pattern
| substitute
Search for the pattern "second"
This output is sent to standard output rather than replacing the file's contents. The result looks like this:
$ sed '/second/s/should/will/' inputfile2
hello world
second line will be replaced
this line should be replaced later
The sed command is case-sensitive. The following won’t work when you try to replace World with there:
$ echo "Hello World" | sed 's/world/there/'
Hello World
GNU sed introduced a new flag, /I, which ignores the case and will perform the replacement with the same command:
$ echo "Hello World" | sed 's/world/there/I'
Hello there
Example: Print a range of lines and quit
With sed, you can also print lines and quit after your criteria are met. The following commands will print three lines and quit. This command:
$ sed -n '1,3p' /etc/passwd
is equivalent to:
$ sed '3q' /etc/passwd
The following would be wrong:
$ sed '1,3q' /etc/passwd # Wrong. You cannot quit three times
Example: Comment out uncommented lines
Regular expressions can also be used with sed, as demonstrated earlier. For example, you have the following small script:
$ cat test_script
#/usr/bin/env bash
this is the first comment
This is another comment
# this is a comment too
echo "This is not a comment and should be echoed"
You now have to skip the first line, starting with #!/bin/bash, and comment out the third and fourth lines, but not the fifth because that line is already commented.
In sed, you can use something like the following:
$ sed '3,6s/^[^#]/# &/g' test_script
#/usr/bin/env bash
# this is the first comment
# This is another comment
# this is a comment too
echo "This is not a comment and should be echoed"
In the above command, the following is performed:
3,6sdefines a range, from line three down to line six./^[^#]/matches everything that is a character and does not start with a hash (#)./# &/greplaces a part, in this case it puts a#in front of the line dictated by the&sign.
Example: Remove all digits
Different applications generate data in different formats. With sed, you can keep only the data you can use. For example, you have the following file (inputfile3) in this format:
foo1234
bar99128
baz2842
qux12953
discard39120
Maybe a program generated the wrong format, or it concatenated the fields to one. What if you were only interested in keeping the alpha characters and wanted to discard the digits? How would you achieve this goal with sed?
The answer is probably easier than you think:
$ sed 's/\([a-z]*\).*/\1/' inputfile3
foo
bar
baz
qux
discard
Example: Change specific lines
Furthermore, sed can also handle ranges by pattern, which means you can specify a start and an end string and manipulate the range. For example:
$ cat inputfile4
hello world
start of the comment
another comment
end of a comment
dont comment this line
nor this line
The following sed command will comment lines starting with start and ending with end:
$ sed '/start/,/end/ s/^/# /' inputfile4
hello world
# start of the comment
# another comment
# end of a comment
dont comment this line
nor this line
Get rid of the empty lines as well.
$ sed '/start/,/end/ s/^/# /;/^$/d' inputfile4
hello world
# start of the comment
# another comment
# end of a comment
dont comment this line
nor this line
There is much more to sed and its rich features. To be able to fully utilize sed’s abilities, please see its documentation page, which you can find here. Also, a great source of information on sed can be found here.
Wrap-up
As I covered previously, you'll use grep when you want to search for a pattern, either in a file or multiple directories recursively. Use sed if you are receiving data from a pipeline, or want to manipulate data on the fly.
The sed command is scripted and it’s easy to learn to perform basic operations. All you need is practice, especially with regular expressions.
