Manipulating text at the command line with sed
In a previous article, I covered how to manipulate text with
grep. Now, turn your attention to the
sed (Stream Editor), which is best suited to be used in pipelines (data that comes from a pipe). The
sed utility can be used to print the contents of a file, substitute a line (or multiple lines), and then save the file. In contrast to
sed can substitute a line or multiple lines in a file and perform an in-place update of that file.
sed invocation when substituting
$ sed 's/foo/bar/' inputfile
Example: Remove comments
grep can format output on screen, it is unable to modify a file in place. To do this, you’d need a file editor like
ed . Since
ed is not part of this article, use
sed to achieve the same thing you did with
grep in the previous article's first example. This time modify the
/etc/fstab file in-place passing the
-i flag to
sed . Without the
-i flag , you'd only see what would have been modified.
You are encouraged to always run
sed without the
-i flag, just to make sure that the result it produces is expected. The
sed utility also offers the
-i.bak flag, which creates a backup file before editing.
grep command for this example was:
$ grep -v '^#' /etc/fstab > ~/fstab_without_comment
sed, you have:
# sed -i '/^#/d' /etc/fstab /dev/mapper/VGCRYPTO-ROOT / ext4 defaults,x-systemd.device-timeout=0 1 1 UUID=e9de0f73-ddddd-4d45-a9ba-1ffffa /boot ext4 defaults 1 2 LABEL=SSD_SWAP swap swap defaults 0 0
Example: Print only
grep example, you printed only usernames from the
/etc/passwd file with the following:
$ grep -Eo '^[a-zA-Z_-]+' /etc/passwd
You can do the same using
sed as follows:
$ sed 's/^\([a-zA-Z_-]\+\).*/\1/' /etc/passwd
In the above example, you group a match by parentheses
(), and then print the matched group with
\1 (back-reference), which dictates the first group. For a second group, you’d use
\2, and so on.
Example: Replace all
sed, you can search for a pattern and then replace only the occurrence matching the pattern. To replace all occurrences in the file
bar globally, run:
$ sed -i '/foo/bar/g' inputfile1
Example: Replace a single instance
Take the file
inputfile2, which has the following contents:
hello world second line should be replaced this line should be replaced later
Say that you want to replace
will, but only for the second line. This command breaks down as follows:
$ sed '/second/s/should/will/' inputfile2 | | | | | | | with this pattern | | this pattern | substitute Search for the pattern "second"
This output is sent to standard output rather than replacing the file's contents. The result looks like this:
$ sed '/second/s/should/will/' inputfile2 hello world second line will be replaced this line should be replaced later
sed command is case-sensitive. The following won’t work when you try to replace
$ echo "Hello World" | sed 's/world/there/' Hello World
sed introduced a new flag,
/I, which ignores the case and will perform the replacement with the same command:
$ echo "Hello World" | sed 's/world/there/I' Hello there
Example: Print a range of lines and quit
sed, you can also print lines and quit after your criteria are met. The following commands will print three lines and quit. This command:
$ sed -n '1,3p' /etc/passwd
is equivalent to:
$ sed '3q' /etc/passwd
The following would be wrong:
$ sed '1,3q' /etc/passwd # Wrong. You cannot quit three times
Example: Comment out uncommented lines
Regular expressions can also be used with
sed, as demonstrated earlier. For example, you have the following small script:
$ cat test_script #/usr/bin/env bash this is the first comment This is another comment # this is a comment too echo "This is not a comment and should be echoed"
You now have to skip the first line, starting with
#!/bin/bash, and comment out the third and fourth lines, but not the fifth because that line is already commented.
sed, you can use something like the following:
$ sed '3,6s/^[^#]/# &/g' test_script #/usr/bin/env bash # this is the first comment # This is another comment # this is a comment too echo "This is not a comment and should be echoed"
In the above command, the following is performed:
3,6sdefines a range, from line three down to line six.
/^[^#]/matches everything that is a character and does not start with a hash (
/# &/greplaces a part, in this case it puts a
#in front of the line dictated by the
Example: Remove all digits
Different applications generate data in different formats. With
sed, you can keep only the data you can use. For example, you have the following file (
inputfile3) in this format:
foo1234 bar99128 baz2842 qux12953 discard39120
Maybe a program generated the wrong format, or it concatenated the fields to one. What if you were only interested in keeping the alpha characters and wanted to discard the digits? How would you achieve this goal with
The answer is probably easier than you think:
$ sed 's/\([a-z]*\).*/\1/' inputfile3 foo bar baz qux discard
Example: Change specific lines
sed can also handle ranges by pattern, which means you can specify a start and an end string and manipulate the range. For example:
$ cat inputfile4 hello world start of the comment another comment end of a comment dont comment this line nor this line
sed command will comment lines starting with start and ending with end:
$ sed '/start/,/end/ s/^/# /' inputfile4 hello world # start of the comment # another comment # end of a comment dont comment this line nor this line
Get rid of the empty lines as well.
$ sed '/start/,/end/ s/^/# /;/^$/d' inputfile4 hello world # start of the comment # another comment # end of a comment dont comment this line nor this line
There is much more to
sed and its rich features. To be able to fully utilize
sed’s abilities, please see its documentation page, which you can find here. Also, a great source of information on
sed can be found here.
As I covered previously, you'll use
grep when you want to search for a pattern, either in a file or multiple directories recursively. Use
sed if you are receiving data from a pipeline, or want to manipulate data on the fly.
sed command is scripted and it’s easy to learn to perform basic operations. All you need is practice, especially with regular expressions.