Ask an OpenShift Admin Office Hour - Compliance and security

Security and compliance are two hot topics when it comes to container-based applications. In this episode, we take a look at why that is and what it means to you as well as how Red Hat OpenShift can help make your software more reliable and more secure. We are joined by three guests today, Doron Caspin and Mark Russel, from the OpenShift product management team, and Juan Osorio Robles, a.k.a. Ozz, from the engineering team.

Our guests delve into how the compliance Operator, in conjunction with the file integrity Operator, lets OpenShift Container Platform admins describe the desired compliance state of a cluster and provides them with an overview of gaps and ways to remediate them. In addition, we spent some time discussing the intrinsic features of Red Hat CoreOS, like FIPS mode, to help bring security both to the platform and applications.

As always, please see the list below for additional links to specific topics, questions, and supporting materials for the episode!

If you’re interested in more streaming content, please subscribe to the Red Hat livestreaming calendar to see the upcoming episode topics and to receive any schedule changes. If you have questions or topic suggestions for the Ask an OpenShift Admin Office Hour, please contact us via Discord, Twitter, or come join us live, Wednesdays at 11am EDT / 1500 UTC, on YouTube and Twitch.

Episode 39 recorded stream:

Use this link to jump directly to where we start talking about today’s topic. 

This week’s top of mind topics:

• Our first topic this week is a recent update to OpenShift which reduces the amount of time cluster Operators take to apply updates. This is the result of some changes to the Operators which enable them to update more instances of the DaemonSet at the same time. For example, the Multus DaemonSet has been configured to update 10% of the nodes at the same time.
• The integrated load balancer used by on-premises IPI deployments cannot be deployed with UPI or non-integrated clusters. At least, not in a supported way outside of two scenarios: OpenShift on OpenStack UPI and when using the Assisted Installer.
• The third topic discussed is about the Machine Config Operator reporting its status during updates and upgrades. Previously, the Operator’s status would report as complete before the Machine Config Pools were done updating, which could cause confusion for administrators and users.
• We moved to discussing a recent BugZilla where the cluster was reporting a “SystemMemoryExceedsReservation” error. This is the result of the core Kubernetes and operating system processes exceeding the amount of memory which has been reserved by the system for them. By default, this is 1 GiB, however it may not be enough. One option to resolve the issue is to allow the systems to automatically allocate resources for this purpose. We’ve talked about this before, but did not dig into the details - but, this time we did! The script which applies the sizing algorithm can be seen here, and if you want to see how that translates into resources reserved, see this section of the stream.
• The last topic covered is about how to determine what changes will result in what actions for configuration changes. Unfortunately, it’s not all documented, but you can see directly in the Operator code what will happen. For example, changing the registries.conf file will result only in CRI-O being restarted, without rebooting the nodes.

Questions answered and topics discussed during the stream:

• Security is a big topic with a lot of different aspects. There’s no one session or live stream that can cover all those, but there are materials which can help. The Red Hat OpenShift security eBook is a great starting point, in addition to the newly released An Open Approach to Vulnerability Management paper.
• Doron explains the role of the compliance Operator and how it fits into a security strategy. The short version is that many organizations have a set of configuration standards and practices which they must adhere to. The compliance Operator is a method of auditing your OpenShift deployment to determine if it is compliant with those standards.
• The compliance Operator translates those configuration settings, as defined by sources like OpenSCAP, into Kubernetes and OpenShift configuration.
• Remediations are defined as a part of the OpenSCAP definition, which could be bash, Ansible, or other ways of automating. But, Kubernetes is not one of those. This is one of the roles of the team, and can include OpenShift specific things like Machine Configs for controlling RHEL CoreOS configuration as well.
• How does GitOps and the compliance Operator work together?
• What does it mean to say that RHEL CoreOS is “immutable”? Can’t I modify things on the system? RHEL CoreOS has a read-only usr file system, but other file systems - etc and var, on the other hand, are read-write. This is necessary for functionality like container images and ephemeral space, along with system configuration.
• Machine Configuration Operator doesn’t validate all files on the entire system. It only cares about the files which it is responsible for controlling. And, it will only notice those changes when it does a configuration remediation task, such as during an update.
• In contrast, the file integrity Operator creates a catalog of all files and tracks if they have been changed for any reason.
• Does a RHEL CoreOS update, which uses rpm-ostree, reset files that are modified outside the OpenShift paradigm? No, the updates only apply to files in the usr file system. You would want to use a combination of the file integrity Operator and Machine Config to find and manage those configurations.
• RHEL CoreOS, as the name implies, is RHEL - just with a reduced set of libraries to focus on being a container and Kubernetes-centric operating system. What about security features? RHEL’s security features, like SELinux, are absolutely a core part of RHEL CoreOS. For example, there are policies which isolate system containers from other containers.
• FIPS is an option when deploying OpenShift. What does that mean? When enabled, FIPS validated crypto modules and others are deployed and used in the system. Using these with UBI-based containers means that the application has a FIPS validated set of policies, modules, and libraries.