More stupid Bash tricks: Variables, find, file descriptors, and remote operations

This blog post is the second of two covering some practical tips and tricks to get the most out of the Bash shell. In part one, I covered history, last argument, working with files and directories, reading files, and Bash functions. In this segment, I cover shell variables, find, file descriptors, and remote operations.

Use shell variables

The Bash variables are set by the shell when invoked. Why would I use hostname when I can use $HOSTNAME, or why would I use whoami when I can use $USER? Bash variables are very fast and do not require external applications.

These are a few frequently-used variables:

$PATH
$HOME
$USER
$HOSTNAME
$PS1
..
$PS4

Use the echo command to expand variables. For example, the $PATH shell variable can be expanded by running:

$> echo $PATH

[ Download now: A sysadmin's guide to Bash scripting. ]

Use the find command

The find command is probably one of the most used tools within the Linux operating system. It is extremely useful in interactive shells. It is also used in scripts. With find I can list files older or newer than a specific date, delete them based on that date, change permissions of files or directories, and so on.

Let's get more familiar with this command.

To list files older than 30 days, I simply run:

$> find /tmp -type f -mtime +30

To delete files older than 30 days, run:

$> find /tmp -type f -mtime +30 -exec rm -rf {} \;

or

$> find /tmp -type f -mtime +30 -exec rm -rf {} +

While the above commands will delete files older than 30 days, as written, they fork the rm command each time they find a file. This search can be written more efficiently by using xargs:

$> find /tmp -name '*.tmp' -exec printf '%s\0' {} \; | xargs -0 rm

I can use find to list sha256sum files only by running:

$> find . -type f -exec sha256sum {} +

And now to search for and get rid of duplicate .jpg files:

$> find . -type f -name '*.jpg' -exec sha256sum {} + | sort -uk1,1 

Reference file descriptors

In the Bash shell, file descriptors (FDs) are important in managing the input and output of commands. Many people have issues understanding file descriptors correctly. Each process has three default file descriptors, namely:

Now that you know what the default FDs do, let's see them in action. I start by creating a directory named foo, which contains file1.

$> ls foo/ bar/
ls: cannot access 'bar/': No such file or directory
foo/:
file1

The output No such file or directory goes to Standard Error (stderr) and is also displayed on the screen. I will run the same command, but this time use 2> to omit stderr:

$> ls foo/ bar/ 2>/dev/null
foo/:
file1

It is possible to send the output of foo to Standard Output (stdout) and to a file simultaneously, and ignore stderr. For example:

$> { ls foo bar | tee -a ls_out_file ;} 2>/dev/null
foo:
file1

Then:

$> cat ls_out_file
foo:
file1

The following command sends stdout to a file and stderr to /dev/null so that the error won't display on the screen:

$> ls foo/ bar/ >to_stdout 2>/dev/null
$> cat to_stdout
foo/:
file1

The following command sends stdout and stderr to the same file:

$> ls foo/ bar/ >mixed_output 2>&1
$> cat mixed_output
ls: cannot access 'bar/': No such file or directory
foo/:
file1

This is what happened in the last example, where stdout and stderr were redirected to the same file:

    ls foo/ bar/ >mixed_output 2>&1
             |          |
             |          Redirect stderr to where stdout is sent
             |                                                        
             stdout is sent to mixed_output

Another short trick (> Bash 4.4) to send both stdout and stderr to the same file uses the ampersand sign. For example:

$> ls foo/ bar/ &>mixed_output

Here is a more complex redirection:

exec 3>&1 >write_to_file; echo "Hello World"; exec 1>&3 3>&-

This is what occurs:

• exec 3>&1                      Copy stdout to file descriptor 3
• > write_to_file              Make FD 1 to write to the file
• echo "Hello World"     Go to file because FD 1 now points to the file
• exec 1>&3                      Copy FD 3 back to 1 (swap)
• Three>&-                       Close file descriptor three (we don't need it anymore)

Often it is handy to group commands, and then send the Standard Output to a single file. For example:

$> { ls non_existing_dir; non_existing_command; echo "Hello world"; } 2> to_stderr
Hello world

As you can see, only "Hello world" is printed on the screen, but the output of the failed commands is written to the to_stderr file.

Execute remote operations

I use Telnet, netcat, Nmap, and other tools to test whether a remote service is up and whether I can connect to it. These tools are handy, but they aren't installed by default on all systems.

Fortunately, there is a simple way to test a connection without using external tools. To see if a remote server is running a web, database, SSH, or any other service, run:

$> timeout 3 bash -c ‘</dev/tcp/remote_server/remote_port’ || echo “Failed to connect”

For example, to see if serverA is running the MariaDB service:

$> timeout 3 bash -c ‘</dev/tcp/serverA/3306’ || echo “Failed to connect”

If the connection fails, the Failed to connect message is displayed on your screen.

Assume serverA is behind a firewall/NAT. I want to see if the firewall is configured to allow a database connection to serverA, but I haven't installed a database server yet. To emulate a database port (or any other port), I can use the following:

[serverA ~]# nc -l 3306

On clientA, run:

[clientA ~]# timeout 3 bash -c ‘</dev/tcp/serverA/3306’ || echo “Failed”

While I am discussing remote connections, what about running commands on a remote server over SSH? I can use the following command:

$> ssh remotehost <<EOF  # Press the Enter key here
> ls /etc
EOF

This command runs ls /etc on the remote host.

I can also execute a local script on the remote host without having to copy the script over to the remote server. One way is to enter:

$> ssh remote_host 'bash -s' < local_script

Another example is to pass environment variables locally to the remote server and terminate the session after execution.

$> exec ssh remote_host ARG1=FOO ARG2=BAR 'bash -s' <<'EOF'
> printf %s\\n "$ARG1" "$ARG2"
> EOF
Password:
FOO
BAR
Connection to remote_host closed.

There are many other complex actions I can perform on the remote host.

Wrap up

There is certainly more to Bash than I was able to cover in this two-part blog post. I am sharing what I know and what I deal with daily. The idea is to familiarize you with a few techniques that could make your work less error-prone and more fun.

[ Want to test your sysadmin skills? Take a skills assessment today. ]