Troubleshooting with session recording in Red Hat Enterprise Linux 8

Seeing a demo of Session Recording in Red Hat Enterprise Linux 8 was an eye-opener for me because it may change the way we manage systems. Let's take a look at what you can do with session recording and where it comes in handy for troubleshooting.

Session recording can help system administrators track and trace system changes. For example, when there is a system upgrade or update, the command lines and output can be recorded accurately with no copy-and-paste needed, or when an issue happens, the recording can be played back to navigate to the root cause.

It can also be useful to troubleshoot an issue in daily operations. Due to some limitations (e.g. network), a remote session or assistance may be not feasible at that moment. In this case, we can follow these steps to use session recording to get additional help:

• Commands to execute are sent to a system administrator by an expert.

• The administrator enables session recording and runs the commands.

• The administrator exports session recording and sends it back to the expert.

• The expert plays back the session and troubleshoots.

Now, let’s see how to export a session. First, please make sure the Web Console (cockpit) and session recording packages have been installed. To do that, run rpm -aq | egrep 'tlog|cockpit-session-recording' with sudo or as root. You should see output like this:

If any of the packages are missing, please follow the guide from Getting started with session recording in Red Hat Enterprise Linux 8 Beta to install them. Additionally, please ensure the session recording is enabled. Those steps are also in the getting started post.

By default, the data of session recordings is written to the journal. But the system cleans up the journal at reboot. Therefore, to save the old recordings, you need to set persistent storages for the systemd journal.

Refer to How to enable persistent logging for the systemd journal.

We also need the package systemd-journal-remote to save journal entries to a file with journal format. To install the package, use yum install systemd-journal-remote as root.

After the above steps, it is ready to record a CLI session. First, login as a user and run some commands for recording. When logged in, you should be able to see the warning message:

When finished, you can run the below command to export the whole journal file with the systemd-journal-remote command which was just installed:

# journalctl -o export | /usr/lib/systemd/systemd-journal-remote -o /tmp/example.journal -

Note: please notice there is a hyphen mark (-) at the end of the command.

With the export file, you need to copy the journal file to another server and put it under /var/log/journal. Go to the Web Console and enter session recording. There you can see the session recording from the server.

From the highlighted area, you can see the session is recorded for user “edward” on host “rhel8beta”, and the detailed time range. By clicking the recording, you could enter the selected recording to play it.

In the above journal file, it will include all the session recordings. So, if you would like to have a specific session recording only, you can do that by specifying a TLOG_REC ID. First, use one of the below three methods which is convenient to you to find out the session ID:

Method 1: Search keyword "rec" to navigate session ID:

# journalctl -o verbose | grep -i \"rec\"

Method 2: Search user ID of tlog to navigate the session ID

# id tlog
# journalctl -xe -o json-pretty _UID=<User ID of tlog>

Method 3: Find the session ID from the Web Console

With the session ID, export the specific journal entries of the session to a journal file:

# journalctl -o export TLOG_REC=<Session ID> | /usr/lib/systemd/systemd-journal-remote -o /tmp/example.journal -

Next, you can directly play the session back from the file via CLI. The session ID is still required when running the below command:

# tlog-play -r journal --file-path=/tmp/example.journal -M TLOG_REC=<Session ID>

As a summary, please see an example about how to export a session with session ID, and play it back from a journal file:

# journalctl -o export TLOG_REC=84ff48bed23944c78bb252edfc1e0815-3bc8-10572ba | /usr/lib/systemd/systemd-journal-remote -o /tmp/example.journal -

# tlog-play -r journal --file-path=/tmp/example.journal -M TLOG_REC=84ff48bed23944c78bb252edfc1e0815-3bc8-10572ba

A little tip here is that the content on the player screen is available to select and copy. Then, you can paste it anywhere is needed. Isn’t that convenient?

That is how to work with session recording in RHEL 8, which is a new feature in RHEL 8. It can be helpful for system administrators and support engineers especially. They can ask a user or customer to record their sessions, reproduce the problem and then export the recorded session to send to an expert or attach to a support case in order to solve.