Even though it took me a long time to get motivated, I finally containerized several personal Linux services. I've documented the project in this series. In this article, we'll take you through a final example, Request Tracker.
To kick off, we looked at some general principles for migrating applications to containers. Then we looked at the containerization of WordPress, and next, we discussed moving MediaWiki into a container. That project was a bit more involved than the first, with the addition of task scheduling. In this final article, we're going to consider a much more complex migration. Specifically, we'll look at Request Tracker. This service might be the most tricky because both the build and run are fairly sophisticated.
Editor's Note: For the purpose of this article, we assume you'll be building your containers on Red Hat Enterprise Linux 8 using podman build. You may be able to use the instructions on other distributions or with other toolchains, however, some modifications may be required.
Moving Request Tracker
Unlike WordPress and MediaWiki, which run on a single-layered image on top of a base image, Request Tracker uses two layers on top of a base image. Let's look at each layer and see why we did it this way.
The first layer is built quite similar to the
httpd-php image. This image adds the necessary services for a Perl-based web application. We include Apache, the FastCGI module, Perl, MariaDB, cron, and some basic utilities for troubleshooting:
FROM registry.access.redhat.com/ubi8/ubi-init MAINTAINER fatherlinux <email@example.com> RUN yum install -y httpd mod_fcgid perl mariadb-server mariadb crontabs cronie iputils net-tools; yum clean all RUN systemctl enable mariadb RUN systemctl enable httpd RUN systemctl enable postfix RUN systemctl disable systemd-update-utmp.service ENTRYPOINT ["/sbin/init"] CMD ["/sbin/init"]
The second layer is where things get pretty sophisticated. Request Tracker uses a lot of Perl modules from CPAN. Many of these modules are compiled with
gcc and take a long time to install. It also took a lot of work to nail down all of these dependencies to get Request Tracker to install successfully. Historically, we would have captured this in a script somewhere, but with containers, we can have it all in one Containerfile. It's very convenient.
[ You might also enjoy: 6 guides on making containers secure ]
The next thing you should notice about this file is that it's a multi-stage build. Podman and Buildah can absolutely do multi-stage builds, and they can be extremely useful for applications like Request Tracker. We could have bind-mounted in directories, as we did with WordPress and MediaWiki, but we chose a multi-stage build instead. This will give us portability and speed if we need to rebuild the application somewhere else.
Multi-stage builds can be thought of as capturing the development server and the production server in a single build file. Historically, development servers were actually the hardest to automate. Since the early days of CFEngine in the mid-1990s, developers refused to use version control and added anything they wanted to development servers to make them work. Often, they didn't even know what they added to make a build complete. This was actually rational when you had long-lived servers that were well backed up, but it always caused pain when systems administrators had to "upgrade the dev server." It was a nightmare to get builds to function on a brand new server with a fresh operating system.
With multi-stage builds, we capture all of the build instructions and even cache layers that are constructed. We can rebuild this development virtual server anywhere we like.
The second stage in this multi-stage build constructs the virtual production server. By splitting this into a second stage, we don't have to install development tools like
expat-devel in the final production image. This reduces our image's size and reduces the size of the software supply chain in network-exposed services. This also potentially reduces the chances of somebody doing something nasty with our container, should they hack in.
We only install the mail utilities in this second stage, which defines the second layer of our production image for Request Tracker. We could have installed these utilities in the
httpd-perl layer, but many other Perl applications won't need mail utilities.
Another convenience of multi-stage builds is that we don't have to rebuild all of those Perl modules every time we want to update the Perl interpreter, Apache, or MariaDB for security patches.
Now, like WordPress and MediaWiki, let's take a look at some of the tricks we use at runtime:
[Unit] Description=Podman container – rt.fatherlinux.com Documentation=man:podman-generate-systemd(1) [Service] Type=simple ExecStart=/usr/bin/podman run -i --rm --read-only -p 8081:8081 --name rt.fatherlinux.com \ -v /srv/rt.fatherlinux.com/code/reminders:/root/reminders:ro \ -v /srv/rt.fatherlinux.com/config/rt.fatherlinux.com.conf:/etc/httpd/conf.d/rt.fatherlinux.com.conf:ro \ -v /srv/rt.fatherlinux.com/config/MyConfig.pm:/root/.cpan/CPAN/MyConfig.pm:ro \ -v /srv/rt.fatherlinux.com/config/RT_SiteConfig.pm:/opt/rt4/etc/RT_SiteConfig.pm:ro \ -v /srv/rt.fatherlinux.com/config/root-crontab:/var/spool/cron/root:ro \ -v /srv/rt.fatherlinux.com/config/aliases:/etc/aliases:ro \ -v /srv/rt.fatherlinux.com/config/main.cf:/etc/postfix/main.cf:ro \ -v /srv/rt.fatherlinux.com/data/mariadb:/var/lib/mysql:Z \ -v /srv/rt.fatherlinux.com/data/logs/httpd:/var/log/httpd:Z \ -v /srv/rt.fatherlinux.com/data/logs/rt4:/opt/rt4/var:Z \ -v /srv/rt.fatherlinux.com/data/backups:/root/.backups:Z \ --tmpfs /etc \ --tmpfs /var/log/ \ --tmpfs /var/tmp \ --tmpfs /var/spool \ --tmpfs /var/lib \ localhost/rt:latest ExecStop=/usr/bin/podman stop -t 3 rt.fatherlinux.com ExecStopAfter=/usr/bin/podman rm -f rt.fatherlinux.com Restart=always [Install] WantedBy=multi-user.target
Like MediaWiki, all of the config files are bind-mounted in read-only, giving us a solid security upgrade. Finally, the data directories are read-write, just like our other containers. One simple observation: We did still bind mount some code into the image for Reminders, which is a small, home-grown set of scripts that send emails and generate tickets for weekly, monthly, and annual entries.
Let's tackle a few last subjects that aren't specific to any one of our containerized Linux services.
Recoverability is something we have to consider carefully. By using
systemd, we get solid recoverability, on par with regular Linux services. Notice
systemd restarts my services without blinking an eye:
podman kill -a 55299bdfebea23db81f0277d45ccd967e891ab939ae3530dde155f550c18bda9 87a34fb86f854ccb86d9be46b5fe94f6e0e15322f5301e5e66c396195480047a C8092df3249e5b01dc11fa4372a8204c120d91ab5425eb1577eb5f786c64a34b
Look at that! Restarted services:
podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 33a8f9286cee localhost/httpd-php:latest /sbin/init 1 second ago Up Less than a second ago 0.0.0.0:80->80/tcp wordpress.crunchtools.com 37dd6d4393af localhost/rt:4.4.4 /sbin/init 1 second ago Up Less than a second ago 0.0.0.0:8081->8081/tcp rt.fatherlinux.com e4cc410680b1 localhost/httpd-php:latest /sbin/init 1 second ago Up Less than a second ago 0.0.0.0:8080->80/tcp learn.fatherlinux.com
Tips and tricks
This is quite useful for making config file changes. We can simply edit the config file on the container host or use something like Ansible and kill all of the containers with the
podman kill -a command. Because we are using
systemd, it will gracefully handle restarting the services. This is very convenient.
It can be tricky to get software to run within a container, especially when you want it to run read-only. You are constraining the process in ways in which it wasn't necessarily designed. As such, here are some tips and tricks.
First, it's useful to install some standard utilities in your containers. In this guide, we installed
net-tools so that we could troubleshoot our containers. For example, with Request Tracker, I had to troubleshoot the following entry in
/etc/aliases, which generates tickets from emails:
professional: "|/opt/rt4/bin/rt-mailgate --queue 'Professional' --action correspond --url http://localhost:8081/"
netstat were all extremely useful because we are also using external DNS and Cloudflare.
Next up is
podman diff, which I used extensively for running containers as read-only. You can run the container in read-write mode and constantly check
podman diff to see what files have changed. Here's an example:
podman diff learn.fatherlinux.com C /var C /var/spool C /var/spool/cron A /var/spool/cron/root C /var/www C /var/www/html A /var/www/html/learn.fatherlinux.com C /root A /root/.backups
Moving to Kubernetes
Notice that Podman will tell us which files have changed since the container started. In this case, every file that we care about is either on a tmpfs or a bind mount. This enables us to run this container as read-only.
Taking a hard look at Kubernetes is a natural next step. Using a command like
podman generate kube will get us part of the way there, but we still need to figure out how to manage persistent volumes and backups on those persistent volumes. For now, we've decided that Podman +
systemd provides a nice foundation. All of the work that we have done with splitting up the code, configuration, and data is requisite to getting us to Kubernetes.
Notes on environment
My environment is a single virtual machine running at Linode.com with 4GB of RAM, two CPUs, and 80GB of storage. I was able to upload my own custom image of RHEL 8 to serve as the container host. Other than setting the hostname and pointing DNS through Cloudflare, I really didn't have to make any other changes to the host. All of the important data is in
/srv, which would make it extremely easy to replace if it were to fail. Finally, the
/srv directory on the container host is completely backed up.
If you are interested in looking at the configuration files and directory structure of
/srv, I have saved the code here in my GitHub.
Like everyone, I have biases, and I think it's fair to disclose them. I served as a Linux Systems Administrator for much of my career before coming to Red Hat. I have a bias towards Linux, and towards Red Hat Enterprise Linux in particular. I also tend toward automation and the psychology of how to make that automaton accessible to regular contributors.
One of my earliest frustrations as a sysadmin was working on a team with 1000 Linux web servers (doing eCards in web 1.0) where documentation for how to contribute to the automation was completely opaque and had no reasoning documented for why things were the way they were. We had great automation, but nobody considered the psychology of how to introduce new people to it. It was sink-or-swim.
This blog aims to help people get over that hump, while at the same time, making it almost self-documenting. I think it's critically important to consider the human inputs and robot outputs of automation. See also: Bootstrapping And Rooting Documentation: Part 1
[ Free cheat sheet: Kubernetes glossary ]
It seems so easy to move a common service like WordPress into containers, but it's really not. The flexible and secure architecture outlined in this article marshals a senior Linux Administrator or Architect's skills to move from a regular LAMP server to OCI-compliant containers. This guide leveraged a container engine called Podman while also preparing your services for Kubernetes. Separating your code, configuration, and data is a requisite step for moving on to Kubernetes. It all starts with solid, foundational Linux skills.
Some decisions highlighted in this article purposefully challenge various misconceptions within the container community—things like using systemd in a container or only focusing on the smallest base image you can find without paying attention to the entire software supply chain. However, the end product is simple to use. It provides a workflow quite similar to a traditional LAMP server, requiring a minimal cognitive load for traditional Linux systems administrators.
Some of the design decisions made in this article are a compromise and imperfect. Still, I made them because I understand both the pressures of a modern DevOps culture and the psychology of operations and development teams. I wanted to provide the flexibility to get more value out of containers. This set of services should be useful as a model for migrating many of your own services into containers. This will simplify their management, upgrade, and recovery. This not only helps existing Linux admins but future cohorts who will inherit these services, including the future version of me who will have forgotten all of the details. These containerized services are essentially self-documenting in a style that is conducive to a successful DevOps culture.
This series is based on "A Hacker's Guide to Moving Linux Services into Containers" on CrunchTools.com and is republished with permission.