Red Hat Enterprise Linux 9.3 on Azure confidential virtual machines: What’s new?

Previously, Red Hat and Microsoft introduced support for Red Hat Enterprise Linux 9.2 (RHEL) on Azure confidential virtual machines (CVMs). The RHEL9.2 CVM Preview image was available as “private preview” and in How to run Red Hat Enterprise Linux 9.2 on Azure confidential virtual machines, I described how to sign up for the preview and get access to the image. With the release of RHEL 9.3 , RHEL CVM Preview image on Azure became available as “public preview” so no specific sign-up process is required. In this article, I will focus on the changes between RHEL 9.2 and RHEL 9.3 for CVM Preview images.

Public preview

With RHEL 9.3 minor release, RHEL CVM Preview image became publicly available in Azure Marketplace. 

The image can run on AMD SEV-SNP CVMs on Azure (DCasv5 and ECasv5 series). The support for Azure CVMs remains a technology preview feature of RHELx 9. The "technology preview'' status of the feature means that currently the RHEL 9.3 CVM Preview image is not intended for production environments, but can be used for non-production workloads.

Changes since RHEL 9.2

RHEL 9.3 Azure CVM image builds on top of RHEL 9.3 release and thus brings all new features and improvements which come with the minor RHEL release. The main CVM-specific update in RHEL 9.3 is the change of the SecureBoot signing scheme for RHEL Unified Kernel Images (UKIs). Traditionally for the x86_64 architecture, RHEL uses the following SecureBoot certificate hierarchy:

RHEL boot process starts with “shim” package which is signed for SecureBoot by Microsoft CA. Shim binary in RHEL carries “Red Hat SecureBoot CA 5” as “vendor certificate” so everything signed by this certificate or its “child” certificates passes SecureBoot check. In RHEL 9.3, we added a dedicated “Red Hat Secure Boot Signing 504” certificate which is exclusively used for signing RHEL UKIs:

To check which certificate was used to sign a Portable Executable (PE) binary, the pesign tool can be used. For example, this is how to check signature on the currently running RHEL UKI:

$ sudo pesign -S -i /lib/modules/`uname -r`/vmlinuz-virt.efi 
---------------------------------------------
certificate address is 0x7f4192d54fc8
Content was not encrypted.
Content is detached; signature cannot be verified.
The signer's common name is Red Hat Secure Boot Signing 504
The signer's email address is secalert@redhat.com
Signing time: Mon Oct 09, 2023
There were certs or crls included.
---------------------------------------------

In this example, the UKI was signed for SecureBoot by the newly introduced “Red Hat Secure Boot Signing 504” certificate.Previously, RHEL UKIs were signed for SecureBoot by the “Red Hat Secure Boot Signing 501” certificate which is used for the traditional RHEL kernel. The main advantage of the new signing scheme for RHEL Azure CVM Preview image is that it significantly simplifies the process of sealing the root volume key for confidential disk encryption.

In our previous article—RHEL confidential virtual machines on Azure: A technical deep dive—we described how data confidentiality at rest is achieved by RHEL 9.2 CVM Preview image. In short, root volume is encrypted and the key to unlock the volume is sealed by the virtual Trusted Platform Module (vTPM) assigned to the particular VM. Sealing a secret to vTPM involves specifying a policy—the conditions which must be met before the secret can be unsealed. 

RHEL 9.2 CVM Preview image used Platform Configuration Registers (PCR) policy which requires the particular values of PCR4 and PCR7 registers to be observed before root volume key can be unsealed:

• PCR7 value proves that the system was booted with SecureBoot enabled and contains the information about the SecureBoot certificates which were used to authorize boot components.
• PCR4 value contains exact hashes of the shim and UKI binaries that were loaded upon boot.

Using PCR4 value along with PCR7 in the policy was necessary to be able to prevent an attack by a malicious or compromised host, which can then try to boot genuine traditional RHEL kernel with a specially crafted initramfs in order to make vTPM extract the root volume key. As both the traditional kernel and UKI were using the same signing SecureBoot certificate, it was not possible to check the fact that a genuine RHEL UKI is loaded by just observing the PCR7 value.

The main disadvantage of using PCR4 value in the sealing policy is the need to re-seal the secret for every new UKI build as the hash of the UKI binary changes. 

With the newly introduced “Red Hat Secure Boot Signing 504” SecureBoot certificate in RHEL 9.3, this is no longer necessary: the certificate is used exclusively for signing RHEL UKIs and thus when measured into PCR7 proves the use of a genuine RHEL UKI. PCR7 value does not normally change with UKI updates.

Note, to get “Red Hat Secure Boot Signing 504” measured into PCR7, the public part of the certificate must be present in the UEFI SecureBoot “db” variable or Machine Owner Key (MOK) list. In case RHEL UKI is booted on a system which only has the standard Microsoft SecureBoot certificates in the UEFI variables, the main “Red Hat Secure Boot CA 5” will be measured instead. Microsoft Azure instances have the correct UEFI profile for RHEL CVM marketplace images, however, some additional configuration may be required for custom VM images.

RHEL 9.3 release brings some other CVM related features. In particular, RHEL9.3 adds support for PCI pass-thru devices to Hyper-V CVMs. Another notable change in the release is that the kexec-tools package gained support for UKIs. This is an important step in making kdump/kexec technologies supported on CVMs. Unfortunately, this cannot be tried yet with the currently existing Azure CVMs as there are pending issues which prevent kexec from working correctly when AMD SEV-SNP technology is in use.

Intel TDX Preview

Microsoft Azure is actively working on expanding the CVM offering. Earlier this year, Intel TDX-based DCesv5-series and ECesv5-series instances were announced in preview. Red Hat and Microsoft teams are actively working together to enable RHEL on these instance types. To get early access to the in-development version of RHEL for Intel TDX instances please fill the following form: https://aka.ms/tdx-rhel-93-preview

Ongoing work

There is ongoing work in a number of upstream projects to streamline the use of CVM-related technologies. For example:

• The Fedora Project is preparing a change in Fedora Linux to simplify UKI usage for various types of deployments. In particular, the use of standard tooling for managing UKIs as well as ARM64 architecture support is expected.
• The Systemd upstream project is working on making remote volume key sealing to the target vTPM a standard feature of systemd-cryptsetup/systemd-cryptenroll tools. Currently, different vendors use custom, non-interoperable solutions and this complicates supporting technologies like Azure’s confidential disk encryption.
• The Linux kernel has recently gained support for Intel TDX VMs on Microsoft Hyper-V hypervisors.

There are multiple pending and merged CVM and UKI related fixes in other projects, including anaconda, dracut, shim and others. 

Conclusion 

The RHEL 9.3 CVM Preview image takes another step towards making Azure Confidential VMs a fully supported platform for RHEL. We are working closely with Microsoft to make sure RHEL is a first class citizen on the currently existing and future Azure CVM instances. Starting with this minor release, RHEL CVM Preview image on Azure is publicly available. We encourage everyone who’s interested in running their workload on CVMs to try it and report issues to Red Hat Jira. Any feedback would be greatly appreciated!