Setting up containerized SSH servers for session recording with tlog

In the first part of this series, I highlighted some of the benefits of session recording and the use of containerized SSH servers. I also showcased a high-level architecture. As a preparatory step, one of the servers with RHEL 8.2 was configured to build and host the SSH server's rootless containers.

After preparing the server and creating the user (tester2), we create a custom Dockerfile for the SSH server with session recording configured.

In the following Dockerfile, we utilize the ubi-init (multi-service) image, which supports systemd and has the CMD /sbin/init as the default to start the systemd init service. The following assumptions are made for this setup.

• Service exposes port 2022 instead of 22 within a container for SSH
• Configure three users (namely tester, tester2, and tester3)
• Use the same UID for container user tester as the user (tester2) running the rootless container on the host so that this user can be root inside the container
• Change UID for tester within the container as 0 (root)
• Configure SSH keys as build arguments for user tester and tester2 and a password-based login for tester3
• Configure session recording for users tester2 and tester3 by setting the login shell as /usr/bin/tlog-rec-session
• Map persistent volumes for /home and /var/log/journal during container creation to retain the users' home directories and SSH service and session recording logs
• Copy the tlog-rec-session.conf from the host into the container with the following configuration for capturing session recording logs using journald:

{"shell":"/bin/bash","notice":"\nATTENTION! Your session is being recorded!\n\n","latency":10,"payload":2048,"log":{"input":false,"output":true,"window":true},"limit":{"rate":16384,"burst":32768,"action":"pass"},"file":{"path":""},"syslog":{"facility":"authpriv","priority":"info"},"journal":{"priority":"info","augment":true},"writer":"journal"}

[ Try out session recording in a live lab: Configuring Terminal Session Recording ]

Dockerfile for SSH container

Here is a sample Dockerfile for the SSH container:

FROM ubi8/ubi-init

ARG ADMIN_PUBLIC_KEY
ARG TESTER2_PUBLIC_KEY

RUN yum -y install openssh-server ed openssh-clients tlog glibc-langpack-en && yum clean all && systemctl enable sshd;
RUN sed -i 's/#Port.*$/Port 2022/' /etc/ssh/sshd_config && chmod 775 /var/run && rm -f /var/run/nologin
RUN mkdir /etc/systemd/system/sshd.service.d/ && echo -e '[Service]\nRestart=always' > /etc/systemd/system/sshd.service.d/sshd.conf

COPY tlog-rec-session.conf /etc/tlog/tlog-rec-session.conf
RUN adduser --system -s /bin/bash -u 1001 tester && \ #UID matching user uid on host
           adduser --system -s /usr/bin/tlog-rec-session -u 1000 tester2 && \
           adduser --system -s /usr/bin/tlog-rec-session -u 1002 tester3 -p y5utxaxUDNFII && \
           mkdir -p /home/tester/.ssh /home/tester2/.ssh /home/tester3/.ssh

RUN touch /home/tester/.ssh/authorized_keys /home/tester2/.ssh/authorized_keys /home/tester3/.ssh/authorized_keys && \
           chmod 700 /home/tester/.ssh /home/tester2/.ssh /home/tester3/.ssh && \
           chmod 600 /home/tester/.ssh/authorized_keys /home/tester2/.ssh/authorized_keys /home/tester3/.ssh/authorized_keys && \
           sed -i 's/1001/0/g' /etc/passwd && \ #Update UID with root UID
           echo ${ADMIN_PUBLIC_KEY} >> /home/tester/.ssh/authorized_keys && \
           echo ${TESTER2_PUBLIC_KEY} >> /home/tester2/.ssh/authorized_keys && \
           chown -R tester2:tester2 /home/tester2/ && chown -R tester3:tester3 /home/tester3/ && chown -R tester:tester /home/tester
LABEL Description="This ssh server is running systemd with rootless podman"
EXPOSE 2022

CMD ["/sbin/init"]

Save the file in the home directory as sshdockfile. Copy the default /etc/tlog/tlog-rec-session.conf to the home directory.

Build the container image

To build the container image with the desired keys, generate a new pair of keys for the user. Next, export the default and the newly-created public key as variables. Finally, pass them as build arguments using the following. Here is the process:

# ssh-keygen-t ecdsa ; ssh-keygen -t ecdsa -f .ssh/tester2

# export ADMIN_PUBLIC_KEY=`cat /home/test2/.ssh/id_ecdsa.pub`

# export TESTER2_PUBLIC_KEY=`cat /home/test2/.ssh/tester2.pub`

# podman build -t testsshd -f sshdockfile --cgroup-manager=cgroupfs --build-arg ADMIN_PUBLIC_KEY="$ADMIN_PUBLIC_KEY" --build-arg TESTER2_PUBLIC_KEY="$TESTER2_PUBLIC_KEY"

The cgroup-manager argument requires cgroups-v2 and is necessary for the rootless container to use systemd successfully. Podman reports the following error unless this argument is passed:

systemd cgroup flag passed, but systemd support for managing cgroups is not available: OCI runtime error

If the build succeeds, the next step is to run the container with the desired persistent volumes. These volumes (for /var/log/journal and /home) are either pre-created, or Podman creates them dynamically in the volume directory under the default storage location (podman info | grep -i graphroot).

Run the SSH server

As the rootless containers don't support CNI plugins for networking, a preferred higher port (>1024) is mapped to expose the SSH service.

# podman run -d --name=rootless_ssh1 -v ssh1home:/home:Z -v ssh1logs:/var/log/journal:Z -p 33000:2022 --cgroup-manager=cgroupfs localhost/testsshd

Test SSH access

From the host machine, log in to the SSH server using the SSH keys for users tester and tester2, and the password (redhat123) for tester3.

# ssh -l tester -i /home/test2/.ssh/id_ecdsa localhost -p 33000

# ssh -l tester2 -i /home/test2/.ssh/tester2 localhost -p 33000

# ssh -l tester3 localhost -p 33000

A message that the session is being recorded appears for tester2 and tester3.

Validate session recording

After logging as testuser2 and testuser3 and executing some commands, there are tlog entries in the journal log file. As the tester user, run the following command within the container:

# journalctl | grep tlog-rec-session

Each tlog entry has a host ID, a record ID and an associated user. For example:

"host":"0ce2921675b0","rec":"26b55a3aafd94f40b49473ac33bd2c96-42-f99b3","user":"tester2"

These log entries are also accessible from the host using the volume location:

/home/test2/.local/share/containers/storage/volumes/ssh1logs/_data/<containerid>/system.journal

There might be more than one directory under ssh1logs/_data if the previous container(s) has been terminated and a new one replaced it.

Replay a recorded session

The tlog-play command supports replaying a session. Typically the administrator is interested in replaying a specific user session. To achieve that, export the session entries for that particular session from the journal logs and save them into a local file.

Identify a specific record ID and use that to filter all the records with the following awk command:

# journalctl --file .local/share/containers/storage/volumes/ssh1logs/_data/<containerid>/system.journal | awk -F '{"' '$0~/<tlog record id>/{ print "\{\"" $2 }' 2>/dev/null > ~/session_recording.log

# tlog-play -r file -i ~/session_recording.log

This way, administrators can segregate and store the specific session logs for each user session and replay them on demand. These session logs can also be shipped to a central location for auditing purposes. By default, tlog does not capture the standard input for the user, so that sensitive information like passwords is not captured and stored in the logs.

More than one containerized SSH server can be run simultaneously on the host (using different port mappings), with each of the containers having separate user profiles and separate persistent volume mappings. The containers themselves can also be configured as systemd services on the host to start them on boot.

Wrap up

We introduce Red Hat IDM and backend servers (integrated with IDM) in the next part. Host-based access control and centralized sudo will be configured for users/groups from IDM to manage which users can execute what commands on which hosts.

[ Getting started with containers? Check out this free course. Deploying containerized applications: A technical overview. ]