Manipulating text at the command line with sed

"he sed she sed" by Ken Hess
In a previous article, I covered how to manipulate text with grep
. Now, turn your attention to the sed
(Stream Editor), which is best suited to be used in pipelines (data that comes from a pipe). The sed
utility can be used to print the contents of a file, substitute a line (or multiple lines), and then save the file. In contrast to grep
, sed
can substitute a line or multiple lines in a file and perform an in-place update of that file.
The simplest sed
invocation when substituting foo
for bar
is:
$ sed 's/foo/bar/' inputfile
Example: Remove comments
While grep
can format output on screen, it is unable to modify a file in place. To do this, you’d need a file editor like ed
. Since ed
is not part of this article, use sed
to achieve the same thing you did with grep
in the previous article's first example. This time modify the /etc/fstab
file in-place passing the -i
flag to sed
. Without the -i
flag , you'd only see what would have been modified.
You are encouraged to always run sed
without the -i
flag, just to make sure that the result it produces is expected. The sed
utility also offers the -i.bak
flag, which creates a backup file before editing.
The final grep
command for this example was:
$ grep -v '^#' /etc/fstab > ~/fstab_without_comment
With sed
, you have:
# sed -i '/^#/d' /etc/fstab
/dev/mapper/VGCRYPTO-ROOT / ext4 defaults,x-systemd.device-timeout=0 1 1
UUID=e9de0f73-ddddd-4d45-a9ba-1ffffa /boot ext4 defaults 1 2
LABEL=SSD_SWAP swap swap defaults 0 0
Example: Print only /etc/passwd
users
In the grep
example, you printed only usernames from the /etc/passwd
file with the following:
$ grep -Eo '^[a-zA-Z_-]+' /etc/passwd
You can do the same using sed
as follows:
$ sed 's/^\([a-zA-Z_-]\+\).*/\1/' /etc/passwd
In the above example, you group a match by parentheses ()
, and then print the matched group with \1
(back-reference), which dictates the first group. For a second group, you’d use \2
, and so on.
Example: Replace all foo
with bar
In sed
, you can search for a pattern and then replace only the occurrence matching the pattern. To replace all occurrences in the file inputfile1
from foo
to bar
globally, run:
$ sed -i '/foo/bar/g' inputfile1
Example: Replace a single instance
Take the file inputfile2
, which has the following contents:
hello world
second line should be replaced
this line should be replaced later
Say that you want to replace should
with will
, but only for the second line. This command breaks down as follows:
$ sed '/second/s/should/will/' inputfile2
| | | |
| | | with this pattern
| | this pattern
| substitute
Search for the pattern "second"
This output is sent to standard output rather than replacing the file's contents. The result looks like this:
$ sed '/second/s/should/will/' inputfile2
hello world
second line will be replaced
this line should be replaced later
The sed
command is case-sensitive. The following won’t work when you try to replace World
with there
:
$ echo "Hello World" | sed 's/world/there/'
Hello World
GNU sed
introduced a new flag, /I
, which ignores the case and will perform the replacement with the same command:
$ echo "Hello World" | sed 's/world/there/I'
Hello there
Example: Print a range of lines and quit
With sed
, you can also print lines and quit after your criteria are met. The following commands will print three lines and quit. This command:
$ sed -n '1,3p' /etc/passwd
is equivalent to:
$ sed '3q' /etc/passwd
The following would be wrong:
$ sed '1,3q' /etc/passwd # Wrong. You cannot quit three times
Example: Comment out uncommented lines
Regular expressions can also be used with sed
, as demonstrated earlier. For example, you have the following small script:
$ cat test_script
#/usr/bin/env bash
this is the first comment
This is another comment
# this is a comment too
echo "This is not a comment and should be echoed"
You now have to skip the first line, starting with #!/bin/bash
, and comment out the third and fourth lines, but not the fifth because that line is already commented.
In sed
, you can use something like the following:
$ sed '3,6s/^[^#]/# &/g' test_script
#/usr/bin/env bash
# this is the first comment
# This is another comment
# this is a comment too
echo "This is not a comment and should be echoed"
In the above command, the following is performed:
3,6s
defines a range, from line three down to line six./^[^#]/
matches everything that is a character and does not start with a hash (#
)./# &/g
replaces a part, in this case it puts a#
in front of the line dictated by the&
sign.
Example: Remove all digits
Different applications generate data in different formats. With sed
, you can keep only the data you can use. For example, you have the following file (inputfile3
) in this format:
foo1234
bar99128
baz2842
qux12953
discard39120
Maybe a program generated the wrong format, or it concatenated the fields to one. What if you were only interested in keeping the alpha characters and wanted to discard the digits? How would you achieve this goal with sed
?
The answer is probably easier than you think:
$ sed 's/\([a-z]*\).*/\1/' inputfile3
foo
bar
baz
qux
discard
Example: Change specific lines
Furthermore, sed
can also handle ranges by pattern, which means you can specify a start and an end string and manipulate the range. For example:
$ cat inputfile4
hello world
start of the comment
another comment
end of a comment
dont comment this line
nor this line
The following sed
command will comment lines starting with start and ending with end:
$ sed '/start/,/end/ s/^/# /' inputfile4
hello world
# start of the comment
# another comment
# end of a comment
dont comment this line
nor this line
Get rid of the empty lines as well.
$ sed '/start/,/end/ s/^/# /;/^$/d' inputfile4
hello world
# start of the comment
# another comment
# end of a comment
dont comment this line
nor this line
There is much more to sed
and its rich features. To be able to fully utilize sed
’s abilities, please see its documentation page, which you can find here. Also, a great source of information on sed
can be found here.
Wrap-up
As I covered previously, you'll use grep
when you want to search for a pattern, either in a file or multiple directories recursively. Use sed
if you are receiving data from a pipeline, or want to manipulate data on the fly.
The sed
command is scripted and it’s easy to learn to perform basic operations. All you need is practice, especially with regular expressions.

Valentin Bajrami
Valentin is a system engineer with more than six years of experience in networking, storage, high-performing clusters, and automation. He is involved in different open source projects like bash, Fedora, Ceph, FreeBSD and is a member of Red Hat Accelerators. More about me