In this blog:
-
Walkthrough of setting up updates for edge systems running Red Hat Enterprise Linux (RHEL), including kickstart files.
-
Discussion of RHEL for Edge components.
With the release of Red Hat Enterprise Linux (RHEL) 8.3 last November, Red Hat added support for resilient edge computing architectures based on proven features within RHEL. RHEL for Edge creates operating system images that update "atomically" with automatic rollback on failure.
These operating system images provide efficient updates to RHEL systems by delivering only the deltas to an updated operating system image instead of the complete image. RHEL for Edge enables rapid application of security updates and patches with minimum downtime and continuity of operations if updates should fail.
RHEL for Edge components
This post augments the content in the RHEL for Edge technical video and it dives deeper into the configuration needed to showcase the various capabilities built-in to RHEL.
Follow the full instructions to demonstrate the RHEL for Edge capabilities. Whereas the GitHub repository tracks the contents of the RHEL for Edge summit session, this post will discuss in greater detail how all these features are configured and used to achieve the desired outcome.
Blueprints
The blueprint file, which follows Tom's Obvious Minimal Language (TOML) formatting, codifies the contents of the RHEL for Edge operating system image. The blueprint reference describes the various table headers and key/value pairs that appear in this file. TOML files resemble the familiar INI file format. The blueprint contains several important sections. At the top is the meta-data definition for this image including the name, description, and version which is shown in this snippet.
name = "RFE" description = "RHEL for Edge" version = "0.0.1"
Packages, modules, and groups entries define the contents of the operating system image. For example, the packages entity can be repeated to describe the list of packages to be added to the default minimal set included in the image. An example to install all versions of the strace package would look like:
[[packages]] name = "strace" version = "*"
Subsequent sections in the blueprint file describe customizations to the generated operating system image. TCP port 8080 is being opened since the image will host a web server, and a user with administrator privileges is also defined. A hashed password is included in the file, but you can also use a plaintext value.
Operating system images output by image-builder contain a defined set of packages based on the image type being generated. At a minimum, the packages from the comps core group are included.
Image Builder for Red Hat Enterprise Linux
RHEL for Edge includes an image builder backend with a web interface. This service, accessed from either the command line or via a web console plug-in, composes operating system images based on the contents of the blueprint file. The resulting rpm-ostree operating system images are then deployed to the relevant systems using a range of methods.
rpm-ostree operating system image
A simple web server offers the operating system as an expanded rpm-ostree tarball. The rpm-ostree format creates a middle-ground between package-based operating systems and image-based operating systems by enabling atomic image updates with automatic rollback on failure while also allowing persistent configuration in various file system locations including /etc and /var. RHEL 8.4 expands the delivery options to include an OCI container that wraps the rpm-ostree image with a built-in web server to provide content to RHEL for Edge systems. Alternatively, the rpm-ostree content can be packaged in a bootable ISO with a self-launching installer for systems in disconnected environments.
The kickstart file
Configuration of the demo occurs principally within the kickstart file. This file installs the initial rpm-ostree image and then configures the RHEL for Edge system. The various scripts in the repository ensure that the installation is started without operator intervention by creating customized boot ISOs that include both the kickstart file and necessary kernel boot parameters. The remainder of this post walks through the content of the kickstart file, explaining the relevance of each section.
Initial installation
The beginning stanza launches installation of our RHEL for Edge device. Review the lines in the kickstart file below.
5 # set locale defaults for the Install 6 lang en_US.UTF-8 7 keyboard us 8 timezone UTC 9 10 # initialize any invalid partition tables and destroy all of their contents 11 zerombr 12 13 # erase all disk partitions and create a default label 14 clearpart --all --initlabel 15 16 # automatically create xfs partitions with no LVM and no /home partition 17 autopart --type=plain --fstype=xfs --nohome 18 19 # poweroff after installation is successfully completed 20 poweroff 21 22 # installation will run in text mode 23 text 24 25 # activate network devices and configure with DHCP 26 network --bootproto=dhcp --noipv6 27 28 # Kickstart requires that we create default user 'core' with sudo 29 # privileges using password 'edge' 30 user --name=core --groups=wheel --password=edge --homedir=/var/home/core 31 32 # set up the OSTree-based install with disabled GPG key verification, the base 33 # URL to pull the installation content, 'rhel' as the management root in the 34 # repo, and 'rhel/8/x86_64/edge' as the branch for the installation 35 ostreesetup --nogpg --url=http://${HOSTIP}:8000/repo/ --osname=rhel --remote=edge --ref=rhel/8/x86_64/edge
Most of this content is boiler-plate. Lines 5 through 17 ensure fresh disk partitions for our installation. Lines 20 through 26 power down the system after installation, enable a text-based installation, and configure networking, respectively.
Of note, line 30 defines user “core” with system administration privileges. The blueprint file used to generate the rpm-ostree operating system image also defines the user “core.” Line 30 is necessary to ensure that the kickstart installation runs unattended and it does not pause for operator input. Anaconda needs a defined root user or other user to enable fully automated and unattended installation.
Line 35 is where most of the work is done. The “ostreesetup” command reaches out to the defined URL and pulls the rpm-ostree content from the specific reference branch to install the operating system. Beyond this point, various “%post” stanzas configure the remainder of the system. We’ll discuss each “%post” section in more detail.
Automated failover with keepalived
Ensuring continuity of operations requires highly available systems with the ability to take over functionality when a primary system fails. The keepalived feature within RHEL enables that behavior. This post focuses the discussion to specifically what is enabled for the demonstration.
For the demonstration we configure primary and backup RHEL for Edge devices. The primary and backup devices use the virtual router redundancy protocol (VRRP), implemented by keepalived, to govern the ownership of a virtual IP address. The primary edge device has a higher priority and it will own the virtual IP address when it’s active. Review the excerpted lines from the kickstart file here:
43 # parse out boot parameters beginning with "vip" 44 cmdline=`cat /proc/cmdline` 45 params=(${cmdline// / }) 46 for param in "${params[@]}"; do 47 if [[ $param =~ "vip" ]]; then 48 eval $param 49 fi 50 done
Lines 43 through 50 do a little kickstart shell magic to get the kernel boot parameters for configuring keepalived. The specific values being passed are the device’s state and priority. The next snippet writes the configuration for keepalived.
53 cat << EOF > /etc/keepalived/keepalived.conf 54 vrrp_instance RFE_VIP { 55 state $vip_state 56 interface enp1s0 57 virtual_router_id 50 58 priority $vip_priority 59 advert_int 1 60 authentication { 61 auth_type PASS 62 auth_pass edge123 63 } 64 virtual_ipaddress { 65 $VIP_IP/$VIP_MASK 66 } 67 } 68 EOF
Lines 53 through 67 create the configuration file for keepalived. Here we create a VRRP instance called “RFE_VIP” and then map the variables for state and priority into the configuration file. VRRP advertisements are sent once a second, with failovers taking around three seconds or less. The above linked article series for keepalived goes into greater detail on the various parameters and settings. This is a minimal configuration to enable failover to the backup instance and then restoration to the primary instance when it’s functional again.
72 cat << EOF > /etc/sysconfig/keepalived 73 KEEPALIVED_OPTIONS="-D --vrrp" 74 EOF
Lines 72 through 74 tailor the keepalived service to enable both logging and running only the VRRP subsystem. The keepalived service offers greater functionality, but the demonstration is configured to only handle failover from primary to backup edge instances.
77 cat << EOF > /etc/systemd/system/enable-vrrp-protocol.service 78 [Unit] 79 Wants=firewalld.service 80 After=firewalld.service 81 82 [Service] 83 Type=oneshot 84 ExecStart=firewall-cmd --add-rich-rule="rule protocol value='vrrp' accept" --permanent 85 ExecStartPost=firewall-cmd --reload 86 87 [Install] 88 WantedBy=multi-user.target default.target 89 EOF 90 91 systemctl enable enable-vrrp-protocol.service
Lines 77 through 91 are critically important to the proper operation of keepalived. The primary and backup edge devices need to send multicast advertisements to each other to ensure proper governance of the virtual IP address.
By default, VRRP packets are blocked by the firewall. A simple one-shot systemd service unit file is created and then the service is enabled to help ensure that VRRP packets are allowed through the firewall. Since some commands are not executable within the kickstart (e.g., there is no running firewalld service when the kickstart is executed), a simple service that runs once at boot up is created to add the VRRP rules to the firewall.
Keeping rpm-ostree up to date
Operating systems should receive regular updates, whether running at the edge or not. Operating at the edge, however, presents unique challenges since there’s limited infrastructure and network connectivity.
Methods should be devised to provide updates that are non-disruptive to normal operations and fit within the constraints of the operational environment. RHEL for Edge is designed so that a failure during an operating system update triggers an automatic rollback to the prior system state so the edge device is never inoperational.
In disrupted, disconnected, intermittent, and limited-bandwidth environments, alternate schemes should be explored. For example, an edge device could delay updates until the system is in range of supporting infrastructure. A temporary network cable could then be connected and a reboot used to trigger the download and application of the updates.
This section of the kickstart file configures periodic checks for operating system updates and then the automatic application of those updates. This assumes an environment where network connectivity is available to support downloading updated content, staging it, and then automatically upgrading.
102 echo AutomaticUpdatePolicy=stage >> /etc/rpm-ostreed.conf
Line 102 modifies the update policy to stage any downloaded operating system updates that will then be applied at the next reboot.
112 cat > /etc/systemd/system/applyupdate.service << 'EOF' 113 [Unit] 114 Description=Apply Update Check 115 116 [Service] 117 Type=oneshot 118 ExecStart=/bin/sh -c 'if [[ $(rpm-ostree status -v | grep "Staged: yes") ]]; then systemctl --message="Applying OTA update" reboot; else logger "Running latest available update"; fi' 119 EOF 120 121 # This systemd timer activates every minute to check for staged 122 # updates to the operating system 123 cat > /etc/systemd/system/applyupdate.timer <<EOF 124 [Unit] 125 Description=Daily Update Reboot Check. 126 127 [Timer] 128 # activate every minute 129 OnBootSec=30 130 OnUnitActiveSec=30 131 132 #weekly example for Sunday at midnight 133 #OnCalendar=Sun *-*-* 00:00:00 134 135 [Install] 136 WantedBy=multi-user.target 137 EOF
Lines 112 through 137 create a systemd timer and accompanying service that periodically runs to check if updated operating system content has been staged. If so, an automatic reboot is triggered. This approach is great for a demonstration, but it can be disruptive to normal operations. The key takeaway is that systemd timers and services and the underlying tooling for RHEL for Edge allows for a broad range of solutions to ensure systems are up to date.
It’s also important to note that systemd timers enable a virtually unlimited way of specifying when and at what intervals a service executes. Lines 127 through 130 specify execution every thirty seconds after boot, but this can follow the commented example of every Sunday at midnight or almost any desired periodicity. Timer events can also be “splayed,” or distributed over a period of time, to avoid having devices overtaxing available infrastructure when requesting updates.
142 systemctl enable rpm-ostreed-automatic.timer applyupdate.timer
Line 142, the last line in this section, enables not only the applyupdate.timer we just defined, but also a second timer included with RHEL for Edge to check if there’s updates and then stage them per earlier policy settings. By default, the rpm-ostreed-automatic.timer triggers every hour to run a service with a matching name. The timing can be modified to override the default settings.
Image registry policy
In the demonstration, systemd services launch container applications via podman. Policy settings tailor both the image registries that are searched and their security settings.
151 cat > /etc/containers/registries.conf <<EOF 152 [registries.search] 153 registries = ['registry.access.redhat.com', 'registry.redhat.io', 'docker.io'] 154 [registries.insecure] 155 registries = ['${HOSTIP}:5000'] 156 [registries.block] 157 registries = [] 158 EOF
Lines 151 through 158 customize the registry list to mark the self-hosted demonstration registry as insecure, so authentication is not required to download a container image. For obvious reasons, container access should be secured in operational environments.
Rootless, socket-activated container application
This next section is vitally important to the demonstration and it illustrates a great many features of both Podman and systemd sockets, timers, and services. For a deeper explanation of how this all works, please review this excellent article covering podman rootless containers with socket activation. Images from that blog post are excerpted here for clarity.
Systemd sockets, after receiving a request, start a service of the same name. The socket will create a file descriptor for the request and pass that to the underlying service. The service will then in turn process the request and listen for additional requests. To enable socket activated container applications, it’s necessary to use a proxy. This is fully explored in the above linked article, but suffice it to say that podman is not able to accept a file descriptor and then listen for requests. The flow that occurs is illustrated here.
A client request to the systemd socket will trigger the proxy service to be started. The proxy service will then trigger a dependent container service to start and the request will be forwarded to the container service. This flow enables on-demand activation of both the proxy and the container services. This is important for the demonstration of failover. Neither the primary or backup systems are running services until they are needed, acting as “scale-from-zero” services. With a future systemd release, it will be possible to “scale-to-zero” creating a truly serverless experience.
There’s quite a bit of configuration for this in the kickstart file so each section will be broken down and discussed separately.
168 # create systemd user directories for rootless services, timers, 171 mkdir -p /var/home/core/.config/systemd/user/sockets.target.wants 172 mkdir -p /var/home/core/.config/systemd/user/timers.target.wants 173 mkdir -p /var/home/core/.config/systemd/user/multi-user.target.wants
Lines 168 through 173 create needed directories when running rootless system sockets, timers, services. Normally these directories would be created through the “systemctl --user” commands but that’s not possible in the limited shell environment of a kickstart file. To workaround this limitation, directories and softlinks are created directly.
176 cat << EOF > /var/home/core/.config/systemd/user/container-httpd-proxy.socket 177 [Socket] 178 ListenStream=$VIP_IP:8080 179 FreeBind=true 180 181 [Install] 182 WantedBy=sockets.target 183 EOF
Lines 176 through 183 define the socket listener for web requests. Line 178 binds the socket listener to the virtual IP address and port 8080. Since the virtual IP address is only configured for one edge device at a time, the FreeBind
option in line 179 allows the binding to occur even if the IP address is not yet configured for the device. This option is described in the ip(7) man page with the relevant section excerpted here.
IP_FREEBIND (since Linux 2.4)
If enabled, this boolean option allows binding to an IP address that is nonlocal or does not (yet) exist. This permits listening on a socket, without requiring the underlying network interface or the specified dynamic IP address to be up at the time that the application is trying to bind to it.
187 cat << EOF > /var/home/core/.config/systemd/user/container-httpd-proxy.service 188 [Unit] 189 Requires=container-httpd.service 190 After=container-httpd.service 191 Requires=container-httpd-proxy.socket 192 After=container-httpd-proxy.socket 193 194 [Service] 195 ExecStart=/usr/lib/systemd/systemd-socket-proxyd 127.0.0.1:8080 196 EOF
Lines 187 through 196 define the proxy service that accepts the file descriptor from the socket and then listens for additional http requests. The proxy forwards requests to the container web server application on the localhost address. The Requires
and After
directives define the dependencies and ordering so that components are started in the correct order when a request is received.
203 cat > /var/home/core/.config/systemd/user/container-httpd.service <<EOF 204 # container-httpd.service 205 # autogenerated by Podman 3.0.2-dev 206 # Thu May 20 10:16:40 EDT 2021 207 208 [Unit] 209 Description=Podman container-httpd.service 210 Documentation=man:podman-generate-systemd(1) 211 212 [Service] 213 Environment=PODMAN_SYSTEMD_UNIT=%n 214 Restart=on-failure 215 TimeoutStopSec=70 216 ExecStartPre=/bin/rm -f %t/container-httpd.pid %t/container-httpd.ctr-id 217 ExecStart=/usr/bin/podman run --conmon-pidfile %t/container-httpd.pid --cidfile %t/container-httpd.ctr-id --cgroups=no-conmon --replace -d --label io.containers.autoupdate=image --name httpd -p 127.0.0.1: 8080:80 ${HOSTIP}:5000/httpd:prod 218 ExecStartPost=/bin/sleep 1 219 ExecStop=/usr/bin/podman stop --ignore --cidfile %t/container-httpd.ctr-id -t 10 220 ExecStopPost=/usr/bin/podman rm --ignore -f --cidfile %t/container-httpd.ctr-id 221 PIDFile=%t/container-httpd.pid 222 Type=forking 223 EOF
Lines 203 through 223 define the systemd service for the container web server. This content was auto-generated as described in the above linked article using the command podman generate systemd
. These settings were then tailored. Notably, line 217 adds the label io.containers.autoupdate=image
to enable auto update of the container application, discussed in more detail below. Line 218 adds a short pause to ensure that the web server, being launched as a container by podman, is fully up and ready to receive an http request.
240 cat > /var/home/core/.config/systemd/user/podman-auto-update.service <<EOF 241 [Unit] 242 Description=Podman auto-update service 243 Documentation=man:podman-auto-update(1) 244 245 [Service] 246 ExecStart=/usr/bin/podman auto-update 247 248 [Install] 249 WantedBy=multi-user.target default.target 250 EOF 251 252 # This timer ensures podman auto-update is run every minute 253 cat > /var/home/core/.config/systemd/user/podman-auto-update.timer <<EOF 254 [Unit] 255 Description=Podman auto-update timer 256 257 [Timer] 258 # This example runs the podman auto-update daily within a two-hour 259 # randomized window to reduce system load 260 #OnCalendar=daily 261 #Persistent=true 262 #RandomizedDelaySec=7200 263 264 # activate every minute 265 OnBootSec=30 266 OnUnitActiveSec=30 267 268 [Install] 269 WantedBy=timers.target 270 EOF
Lines 240 through 270 define both a system timer and corresponding service to update the container application if the container image in the registry differs from what’s currently running on the edge device. The timer and service periodically run the command podman auto-update
that then does all the work necessary to check if a changed image is in the registry, download the image content, stop the currently running container application, and finally restart the container application with the updated container image.
By default, this service is triggered every thirty seconds after boot but lines 260 through 262 illustrate how this can be run daily within a randomized two hour window so requests to the registry are “splayed.”
273 cat > /var/home/core/.config/systemd/user/pre-pull-container-image.service <<EOF 274 [Service] 275 Type=oneshot 276 ExecStart=podman pull $HOSTIP:5000/httpd:prod 277 278 [Install] 279 WantedBy=multi-user.target default.target 280 EOF
Lines 273 through 280 implement an optimization to pre-pull the container images and speed up starting the container web server application when a request is received. This runs once at boot.
282 # enable socket listener 283 ln -s /var/home/core/.config/systemd/user/container-httpd-proxy.socket /var/home/core/.config/systemd/user/sockets.target.wants/container-httpd-proxy.socket 284 285 # enable timer 286 ln -s /var/home/core/.config/systemd/user/podman-auto-update.timer /var/home/core/.config/systemd/user/timers.target.wants/podman-auto-update.timer 287 288 # enable pre-pull container image 289 ln -s /var/home/core/.config/systemd/user/pre-pull-container-image.service /var/home/core/.config/systemd/user/default.target.wants/pre-pull-container-image.service 290 ln -s /var/home/core/.config/systemd/user/pre-pull-container-image.service /var/home/core/.config/systemd/user/multi-user.target.wants/pre-pull-container-image.service
Lines 282 through 290 manually create the various soft links to enable the rootless systemd sockets, timers, and services to run. This is necessary since it’s not possible to run the systemctl --user
commands during kickstart as there is no logged in user or fully configured shell.
293 chown -R core: /var/home/core 294 restorecon -vFr /var/home/core
Lines 293 through 294 ensure that both discretionary and mandatory access controls are correct for the rootless systemd unit files.
297 cat << EOF > /etc/systemd/system/enable-linger.service 298 [Service] 299 Type=oneshot 300 ExecStart=loginctl enable-linger core 301 302 [Install] 303 WantedBy=multi-user.target default.target 304 EOF 305 306 systemctl enable enable-linger.service
And wrapping this section up, lines 297 through 306 create a service run once at boot to enable linger for user core
. Linger enables rootless systemd sockets, timers, and services to run whether the associated user is logged in or not.
Greenboot and automated rollbacks
RHEL for Edge supports conditional checks to be run during system startup to determine if the current operating system image meets all operational requirements. The requirements are defined in the /etc/greenboot directory and enforced via the greenboot facility, a generic health check framework for systemd. Greenboot is implemented as simple shell scripts in a prescribed directory structure that return pass/fail results. The directory structure is shown below:
/etc/greenboot +-- check | +-- required.d /* these scripts MUST succeed */ | +-- wanted.d /* these scripts SHOULD succeed */ +-- green.d /* scripts run after success */ +-- red.d /* scripts run after failure */
All scripts in the required.d
directory must return a successful result for startup to proceed. If there's three failed boot attempts, the operating system will be rolled back to the previous version and restarted. Scripts within the wanted.d directory may succeed, but they won't trigger a rollback if they fail. The green.d directory contains any scripts that should run as part of a successful boot and the scripts in the red.d directory will run if there’s a failure.
The demonstration uses a custom greenboot check script to control whether a rollback occurs to the previous version of the operating system. This is explained below.
322 mkdir -p /etc/greenboot/check/required.d 323 cat > /etc/greenboot/check/required.d/01_check_upgrade.sh <<EOF 324 #!/bin/bash 325 326 # 327 # This test fails if the current commit identifier is different 328 # than the original commit 329 # 330 331 if [ ! -f /etc/greenboot/orig.txt ] 332 then 333 rpm-ostree status | grep -A2 '^\*' | grep Commit > /etc/greenboot/orig.txt 334 fi 335 336 rpm-ostree status | grep -A2 '^\*' | grep Commit > /etc/greenboot/current.txt 337 338 diff -s /etc/greenboot/orig.txt /etc/greenboot/current.txt 339 EOF 340 341 chmod +x /etc/greenboot/check/required.d/01_check_upgrade.sh
Lines 322 to 339 define the script itself and line 341 ensures its executable by the greenboot facility. At system startup, the file orig.txt
is created only if it doesn’t already exist. This file contains the unique commit identifier of the current operating system image. On each startup, the file current.txt
is overwritten with the latest commit identifier for the current operating system image. The two files are then compared, and if they differ, the greenboot check fails.
This mechanism prevents an upgrade from occurring. During the demonstration, the system will attempt three boots with an updated operating system image and then rollback to the previous image. To enable a successful upgrade, simply remove the orig.txt
file before attempting the upgrade.
Conclusion
This blog article fully describes the configuration of the RHEL for Edge servers supporting the demonstration. This includes the various systemd timers, sockets, and services to enable rootless, on-demand container web server; automated failover from primary to backup edge servers; automated updates of the container web server; and automated operating system image upgrades with rollback on failure. RHEL provides a very complete and extremely lightweight approach to support computing at the edge.
About the author
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit