Ask an OpenShift Admin Office Hour - Red Hat CoreOS

Once an application is deployed, we don’t think about the operating system much - unless it breaks. This is even more true for containerized applications, whether they’re deployed to a single host using Podman or across a Kubernetes cluster. But even though the operating system is mostly ignored by the application team, it plays an important role for administrators and can dramatically affect our experience.

OpenShift 4 introduced a significant shift in how Red Hat deploys and manages the operating system underpinning everything else in OpenShift, changing from Red Hat Enterprise Linux (RHEL), and sometimes RHEL Atomic, to Red Hat Enterprise Linux CoreOS (RHCOS).

This episode we welcome Mark Russell, product manager, and Derrick Ornelas, product experience engineer, to talk about why and how RHCOS is different, along with discussing some configuration options for manageability, performance, and resiliency.

As always, please see the list below for additional links to specific topics, questions, and supporting materials for the episode!

If you’re interested in more streaming content, please subscribe to the OpenShift.tv streaming calendar to see the upcoming episode topics and to receive any schedule changes. If you have questions or topic suggestions for the Ask an OpenShift Admin Office Hour, please contact us via Discord, Twitter, or come join us live, Wednesdays at 11am EDT / 1500 UTC, on YouTube and Twitch.

Episode 28 recorded stream:

Supporting links for today:

• Use this link to jump directly to where we start talking about today’s topic
• We were absent the last two weeks as a result of Red Hat Summit, GitOpsCon, OpenShift Commons, and KubeCon EMEA. We have some great summaries of each of those events at the links, so if you’re curious what we were doing be sure to give them a read!
• Deploying an OpenShift cluster across multiple sites, for example having control plane nodes in three different datacenters, is possible, but there are some important things you should know and prepare for. This KCS provides some guidelines and recommendations if you choose to pursue multi-site OpenShift.
• Updates / upgrades to OpenShift 4.7 are available now! If you have deployed your OpenShift cluster to vSphere using a non-integrated, platform agnostic (a.k.a. bare metal UPI or simply set platform=none in the install-config.yaml), then you’ll want to pay special attention to the known issues! Due to a bug between VXLAN offload and the RHEL 8.3 kernel used by RHCOS in OpenShift 4.7, there can be some packet loss occurring. The workaround in the release notes is to use VM hardware version 13, but if you’re using a newer version you can also apply machine config to disable the offload and workaround the issue that way.
• Did you know that you can change the domain name used for Routes? Setting an appsDomain on the Ingress configuration will cause all Routes to use the configured domain instead of the default. FYI, the docs say it’s AWS only, but that’s incorrect (and there’s a BZ to fix it) - the option is supported with any deployment type!
• Last, but not least, following up from when Katherine Dubé was on the stream to talk about installation methods, there’s now a blog post which shows how to manually add nodes to vSphere IPI clusters.

Questions answered during the stream:

• What is Red Hat Enterprise Linux CoreOS, where did it come from, and what’s different about it from regular RHEL? RHCOS can trace it’s history to Container Linux and Atomic Linux, adapting and integrating the best aspects of both while utilizing a RHEL core.
• How does the management philosophy of RHCOS differ from traditional Linux? RHCOS is more like an appliance, with the configuration being managed via OpenShift.
• What does immutability mean to RHCOS?
• What is Ignition and how does it fit in? Ignition runs before the system pivots and does low level configuration very early in the boot process. For example, it’s able to partition disks or create software RAID devices at installation time.
• How does Ignition at install time compare with Ignition used by MachineConfig? They use the same syntax to describe configuration, but boot-time ignition only runs once - at system install.
• Not all reconfiguration actions done with Ignition and machine config require a reinstall, but are there some that we recommend or encourage administrators to do a reinstall?
• Is it possible, and recommended, to set network configuration day 2+ using machine config and Ignition to directly configure NetworkManager? For secondary+ interfaces, yes, this is one way you can do it, but we would recommend the nmstate Operator in the future when it’s generally available.
• How do I add agents, drivers, and other software that isn’t containerized to the RHCOS nodes? Mark answered this question saying that it’s often up to the third party partner or vendor to support the OpenShift model using DaemonSets. Unfortunately, if they only deploy using an RPM or something similar, there isn’t a supported way of doing this today. One thing that we didn’t mention on the stream is that in some cases it doesn’t make sense or apply to RHCOS. For example, package management agents aren’t applicable. Finally, the Special Resource Operator makes it possible to deploy kernel drivers (kmods) and other functions to the nodes to surface specialized capabilities.
• What is Butane? Not just a flammable gas, Butane is a tool to generate Ignition files to configure a host. This makes it dramatically easier to do things like create mirrored disks for the operating system install.
• How do I, as an administrator, get started with RHCOS?