Based on Kata Containers, the Confidential Containers (CoCo) project is a community solution to enable hardware technologies for virtualized memory encryption in container environments through attestation. CoCo SEV enables an encrypted container launch feature by utilizing a remote key broker service to verify the guest measured environment before releasing the image decryption key during orchestration. This blog demonstrates how to prepare an EPYC™ CPU-powered machine for SEV and CoCo, how to install CoCo using a Kubernetes operator, and how to create an encrypted image and start a container pod that uses CoCo SEV.
Requirements
First and foremost, you must have a machine powered by an AMD EPYC™ 7002 Series processor (or later) with Secure Encrypted Virtualization (SEV).
For this blog, I'm using two machines:
- A Dell Inc. PowerEdge R6515 Server running Fedora Linux 35 and a single-node Kubernetes cluster
- A Fedora 36 (x86_64) workstation used to create the encrypted image for the CoCo workload
This separation is not mandatory. You can build encrypted container images on the cluster nodes if you want too.
Server setup
Here are the required packages on the server:
- git
- pip
- mysql-server
- cargo and rustfmt (used to build simple-kbs)
Install the sev-snp-measure tool so you can measure the guest VM:
$ pip3 install sev-snp-measure
Install Kubernetes, and configure it to use the containerd runtime. For this tutorial, I created a single-node Kubernetes v1.24 cluster with the kubeadm tool.
Workstation setup
Creating an encrypted image doesn't involve specialized encryption hardware, so it can be created on any machine. I used a separate machine for convenience. If you're using a separate workstation to build your containers, then you need these packages:
- git
- docker-ce
- skopeo
- cargo and rustfmt and openssl-devel (used to build the CoCo Keyprovider)
Create an encrypted container image
CoCo can deal with regular unencrypted container images, but this blog focuses on a confidential pod with an encrypted image. This section describes how to create a custom nginx image, encrypt it with Skopeo, and finally push it to a Docker Hub registry for later use as a pod image.
I'm using Docker Hub because it supports storing encrypted images. If you use a different registry, make sure it supports encrypted images.
1. Create an image
First, create a Dockerfile to define a custom NGINX image with a special HTML page:
FROM nginx:latest
RUN echo "Welcome to Confidential Containers World!" >> /usr/share/nginx/html/coco.html
Build it:
$ docker build -t coco-custom-nginx .
You've now created an unencrypted image called coco-custom-nginx
in your local Docker storage.
2. Start the CoCo keyprovider
Skopeo uses the ocicrypt library to encrypt a container image. An encryption key is provided to Skopeo through the CoCo Keyprovider. This is a key provider implementation in the attestation-agent project. Build and start the key provider on localhost, port 50001:
$ git clone https://github.com/containers/attestation-agent
$ cd attestation-agent
$ git checkout v0.5.0 -b v0.5.0
$ cd coco_keyprovider
$ cargo build --release
$ cp target/release/coco_keyprovider .
$ RUST_LOG=coco_keyprovider \
./coco_keyprovider --socket 127.0.0.1:50001 &
The ocicrypt.conf
file contains connection information for the key providers. Create the configuration file and export the OCICRYPT_KEYPROVIDER_CONFIG
variable:
$ cat <<EOF > ocicrypt.conf
{
"key-providers": {
"attestation-agent": {
"grpc": "127.0.0.1:50001"
}}}
EOF
$ export OCICRYPT_KEYPROVIDER_CONFIG="$(pwd)/ocicrypt.conf"
3. Generate an encryption key
The encryption key must have a length of 32 bytes. Generate a random base64 encoded key of the appropriate length:
$ export ENC_KEY_BASE64=$(head -c32 < /dev/random | base64)
$ echo $ENC_KEY_BASE64
hRF3FrDC1LBpWtgVpXL3GyIgxNXGG1f5q4HdeR5kDZU=
Copy the encryption key to a file and export the key file path to the ENC_KEY_FILE
variable:
$ echo $ENC_KEY_BASE64 | base64 -d > key1
$ export ENC_KEY_FILE="$PWD/key1"
4. Encrypt the image
Encrypt the coco-custom-nginx image from your local Docker storage, and then push it to the registry (in this example, wainersm/coco-custom-nginx:encrypted
). The new registry must be created in Docker Hub with your user account.
$ docker login -u <your-username>
$ skopeo copy --insecure-policy --encryption-key \
provider:attestation-agent:keypath=${ENC_KEY_FILE}::keyid=kbs:///default/key/key_id1
docker-daemon:coco-custom-nginx:latest \
docker://<your-username>/coco-custom-nginx:encrypted
Getting image source signatures
...
Copying config 9ef34dd43c done
Writing manifest to image destination
Storing signatures
The skopeo copy
command can use several different transport protocols to copy from origin to the destination image. In this example, I've copied the source image from my local Docker storage using the docker-daemon protocol.
Currently, Skopeo is not able to encrypt an image to local Docker storage. It silently fails, so copy the image directly to the registry, and then verify that it was actually encrypted:
$ skopeo inspect \
docker://<your-username>/coco-custom-nginx:encrypted | \
jq '.LayersData\[\].MIMEType'
"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"
"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"
...
"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"
"application/vnd.oci.image.layer.v1.tar+gzip+encrypted"
From the output, you can see that the mimetypes indicate the layers are indeed encrypted.
Prepare the SEV host system
Secure Memory Encryption (SME) needs to be enabled in the system BIOS settings. Using a PowerEdge R6515 server as an example, the Dell Remote Access Controller (DRAC) can be configured to enable this setting in Configuration > BIOS Settings > Processor Settings > Secure Memory Encryption - Enabled
For SEV support, your host system must have a Linux kernel version 5.11 or later. Ensure the running Kernel Virtual Machine (KVM) module has SEV enabled by adding the kvm_amd.sev=1 parameter to the system kernel command line in the GRUB configuration:
$ sudo grubby --default-kernel \
/boot/vmlinuz-5.16.11-200.fc35.x86_64
$ sudo grubby --args="kvm_amd.sev=1" --update-kernel \
/boot/vmlinuz-5.16.11-200.fc35.x86_64
$ sudo grubby --info /boot/vmlinuz-5.16.11-200.fc35.x86_64
index=4
kernel="/boot/vmlinuz-5.16.11-200.fc35.x86_64"
args="ro rootflags=subvol=root ro rootflags=subvol=root
console=ttyS1,115200 kvm_amd.sev=1"
root="UUID=5304c86d-73a8-44b8-9323-fd4a7e52f75b"
initrd="/boot/initramfs-5.16.11-200.fc35.x86_64.img"
title="Linux 5.16.11-200.fc35.x86_64"
id="a6fa0c7913f646b2a3b842ccfe8e8298-5.16.11-200.fc35.x86_64"
Reboot the machine and verify that the messages in the Kernel ring buffer mention SEV:
$ sudo dmesg | grep -i sev
[ 0.000000] Command line:
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.16.11-200.fc35.x86_64
root=UUID=5304c86d-73a8-44b8-9323-fd4a7e52f75b ro rootflags=subvol=root
ro rootflags=subvol=root console=ttyS1,115200 kvm_amd.sev=1
[ 0.069214] Kernel command line:
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.16.11-200.fc35.x86_64
root=UUID=5304c86d-73a8-44b8-9323-fd4a7e52f75b ro rootflags=subvol=root
ro rootflags=subvol=root console=ttyS1,115200 kvm_amd.sev=1
[ 5.754753] ccp 0000:42:00.1: sev enabled
[ 5.811254] ccp 0000:42:00.1: SEV firmware update successful
[ 6.142119] ccp 0000:42:00.1: SEV API:1.42 build:42
[ 19.973676] SEV supported: 509 ASIDs
1. Generate the certificate chain
The SEV certificate chain file contains the AMD Root Key (ARK) and the AMD SEV Signing Key (ASK) certificates. This certificate chain must be downloaded and saved onto the host by an appropriate authority so that the measurement can be verified during launch attestation.
Use the sevctl tool to download the certificate chain file.
At the time of writing, the Fedora 35 package of sevctl
had a bug that caused the certificate chain file creation to fail. Build the tool from source:
$ git clone https://github.com/virtee/sevctl
$ cd sevctl
$ cargo build --release
$ cp target/release/sevctl .
$ sudo mkdir -p /opt/sev/
$ sudo ./sevctl export --full /opt/sev/cert_chain.cert
2. Install CoCo
The CoCo runtime is bundled in a Kubernetes operator you can deploy on your cluster. You must install the CoCo operator version 0.5.0 or higher.
The cluster admin uses labels to instruct the operator controller about which nodes, in a multi-node cluster, need the runtime. You must have the node-role.kubernetes.io/worker=
label on all cluster nodes that you want the runtime installed on:
# kubectl label node "$(hostname)" "node-role.kubernetes.io/worker="
node/virtlab1012 labeled
# kubectl get nodes
NAME STATUS ROLES AGE VERSION
virtlab1012 Ready control-plane,worker 24m v1.24.0
Once the target worker nodes are properly labeled, the next step is to install the operator controller. For this step, ensure that SELinux is in permissive mode. The operator controller attempts to restart services on your system that SELinux may deny. Use the following sequence of commands to install the operator controller:
# kubectl apply -k \
github.com/confidential-containers/operator/config/release?ref=v0.5.0
This creates a series of resources in the confidential-containers-system
namespace. In particular, it creates a deployment with pods, all of which need to be running before you continue the installation.
# kubectl get pods -n confidential-containers-system
NAME READY STATUS...
cc-operator-controller-manager...5jtsk 2/2 Running...
The operator controller is capable of managing the installation of different CoCo runtimes through Kubernetes custom resources. You must install the runtime that provides the support for AMD SEV:
# kubectl apply -k \
github.com/confidential-containers/operator/config/samples/ccruntime/default?ref=v0.5.0
Next, ensure that all the pods in the confidential-containers-system
namespace are running and the system has the kata-qemu-sev
runtime class configured in the cluster:
# kubectl get pods -n confidential-containers-system
NAME READY STATUS...
cc-operator-controller-manager...5jtsk 2/2 Running...
cc-operator-daemon-install-hvfs7 1/1 Running...
cc-operator-pre-install-daemon-4jhkn 1/1 Running...
# kubectl get runtimeclass
NAME HANDLER AGE
kata kata 4m21s
kata-clh kata-clh 4m21s
kata-clh-tdx kata-clh-tdx 4m21s
kata-qemu kata-qemu 4m21s
kata-qemu-sev kata-qemu-sev 4m21s
kata-qemu-tdx kata-qemu-tdx 4m21s
It's a good idea to create a simple Kata Containers pod to ensure that the installation is working as expected before moving on to the next section. Follow the instructions in the project quickstart.
Configure the Key Broker Server (KBS)
The Confidential Containers simple-kbs is a basic prototype key broker that can validate a guest measurement according to a specified policy. By evaluating these policies, a secret or key can be conditionally released. In this case, the image decryption key is released based on a policy that defines an expected SEV launch measurement. If the measured value during launch matches this expected value in the simple-kbs, the image decryption key is released, the image is decrypted and the launch of the pod proceeds.
Clone and build the simple-kbs:
$ git clone https://github.com/confidential-containers/simple-kbs.git
$ cd simple-kbs
$ cargo build --release
$ cp target/release/simple-kbs .
1. Initialize the simple-kbs database
The simple-kbs relies on a SQL database to store its secrets and policies. It currently supports SQLite, Postgres, and MySQL. For this tutorial, I use a MySQL database.
To set up and install the simple-kbs, environment variables must be specified detailing the database connection information and authentication details:
KBS_DB_TYPE
: Database typeKBS_DB_HOST
: Database address (host:port)KBS_DB_USER
: Database usernameKBS_DB_PW
: Database passwordKBS_DB
: Database name
For example:
$ export KBS_DB_TYPE=mysql
$ export KBS_DB_HOST=localhost
$ export KBS_DB_USER=root
$ export KBS_DB_PW=My_Strong_Passphrase
$ export KBS_DB=sev_attest
Export the encryption key (encoded in base64) used to encrypt the image in the previous steps:
$ export ENC_KEY="hRF3FrDC1LBpWtgVpXL3GyIgxNXGG1f5q4HdeR5kDZU="
The database can be initialized as follows:
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -e "CREATE DATABASE ${KBS_DB};"
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} < db/db-mysql.sql
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} -e "INSERT INTO secrets VALUES (10, 'default/key/key_id1', '${ENC_KEY}', NULL);"
The encryption key (${ENC_KEY}
) used to create the docker.io/<your-username>/coco-custom-nginx:encrypted
image is stored in the secrets
table.
Verify the secret
table:
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} -e "SELECT* FROM secrets;"
mysql: [Warning] Using a password on the command-line interface can be insecure.
+----+---------------------+------------------+-------+
| id | secret_id | secret | polid |
+----+---------------------+------------------+-------+
| 10 | default/key/key_id1 | hRF3Fr...R5kDZU= | NULL |
+----+---------------------+------------------+-------+
Now the database is configured with the minimal policy for simple-kbs to serve the encryption key.
2. Start the simple-kbs
Start simple-kbs on localhost at port 44444:
$ ./simple-kbs --grpc_sock=0.0.0.0:44444 &
SEV runtime configuration
When you deployed the CoCo operator, several Kubernetes Runtime Classes and their Kata configuration files were installed:
$ ls /opt/confidential-containers/share/defaults/kata-containers
configuration-clh-tdx.toml configuration-qemu-se.toml
configuration-qemu-tdx.toml configuration.toml
configuration-clh.toml configuration-qemu-sev.toml
configuration-qemu.toml
configuration-dragonball.toml configuration-qemu-snp.toml
configuration-remote.toml
The kata-qemu-sev
runtime class must be specified to launch the pod with the encrypted image with SEV launch attestation, and its configuration file is configuration-qemu-sev.toml
This contains properties that Kata Containers use to instantiate and manage the pod. For example, if the confidential_guest
property is set, then the SEV feature for VM memory encryption is active.
The sev_cert_chain
property sets the location of the SEV certificate chain that was created in a previous section of this blog.
Run your first confidential pod
For ease of use and demonstration, I'm configuring the simple-kbs with a permissive policy to release the encryption key and allow a container pod to be launched without attestation. To configure the full secure policy, skip to the next section. This section describes how to create a confidential pod without attestation.
Here's my example pod YAML (coco-sev-nginx.yaml
). Replace <your-username>
and <server-IP>
:
$ cat coco-sev-nginx.yaml
apiVersion: v1
kind: Pod
metadata:
name: coco-custom-nginx
annotations:
io.katacontainers.config.pre_attestation.enabled: "true"
io.katacontainers.config.pre_attestation.uri: "<server-IP>:44444"
io.katacontainers.config.sev.policy: "3"
spec:
containers:
- name: nginx
image: <your-username>/coco-custom-nginx:encrypted
ports:
- containerPort: 80
dnsPolicy: ClusterFirst
runtimeClassName: kata-qemu-sev
Apply the configuration:
$ kubectl apply -f coco-sev-nginx.yaml
pod/coco-custom-nginx created
The runtime class must be specified as kata-qemu-sev
. The encrypted image URL is set to <your-username>/coco-custom-nginx:encrypted
. Several kubernetes annotations are set to provide the required configuration:
io.katacontainers.config.pre_attestation.enabled
enables SEV launch attestationio.katacontainers.config.pre_attestation.uri
is the address of the simple-kbs (the system's IP, in my example)io.katacontainers.config.sev.policy
is the hardware policy (3 for SEV and 7 for SEV-ES)
Start the pod:
$ kubectl apply -f coco-sev-nginx.yaml
After a little time, verify that the pod is running:
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
coco-custom-nginx 1/1 Running 0 3m59s
You've just created your first confidential pod using AMD SEV!
Use the guest measurement and verification policy
In the previous section, the minimal simple-kbs policy allowed for release of the encryption key without attestation. This was done as a quick proof-of-concept, and is inherently insecure.
To correctly benefit from the workload encryption key, you must create a policy requiring the guest launch measurement to be verified before release.
When the pod launch is scheduled, the Kata shim layer retrieves the SEV launch measurement from the AMD Platform Security Processor (PSP) and passes it to simple-kbs to be verified against your policy. Data sent to simple-kbs is secured by cryptographic keys and signatures from known, trusted authorities.
The sev-snp-measure
tool can be used to recreate the launch measurement. This measurement includes digest values of the guest firmware, kernel, initrd, and kernel command-line parameters.
Set the paths to the necessary files:
$ PATH=$PATH:/opt/confidential-containers/bin/
$ KATA_SEV_CONFIG="/opt/confidential-containers/share/defaults/kata-containers/configuration-qemu-sev.toml"
$ KERNEL_PATH=$(kata-runtime -config "$KATA_SEV_CONFIG" kata-env --json | jq -r '.Kernel.Path')
$ INITRD_PATH=$(kata-runtime -config "$KATA_SEV_CONFIG" kata-env --json | jq -r '.Initrd.Path')
$ FIRMWARE_PATH=$(grep "^firmware = " $KATA_SEV_CONFIG | sed -e 's/.*"\\.*\\".*/\1/')
The kernel command-line parameters are a bit tricky to obtain because Kata combines the values from the configuration file (configuration-qemu-sev.toml
) with some additional values generated at runtime. One way to obtain it is to grep for the -append
parameter passed to QEMU from the confidential pod youcreated in the previous section:
$ ps aux | grep /opt/confidential-containers/bin/qemu-system-x86_64
root 2656095 99.8 6.7 2565268 21342 ? Sl 14:09 94:39
/opt/confidential-containers/bin/qemu-system-x86_64 -name
...
-kernel
/opt/confidential-containers/share/kata-containers/vmlinuz-5.19.2-102cc-sev
-initrd
/opt/confidential-containers/share/kata-containers/kata-ubuntu-20.04-sev.initrd
-append tsc=reliable no_timer_check rcupdate.rcu_expedited=1
i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp
reboot=k cryptomgr.notests net.ifnames=0 pci=lastbus=0 console=hvc0
console=hvc1 quiet panic=1 nr_cpus=1 selinux=0
agent.aa_kbc_params=online_sev_kbc::10.8.0.194:44444
scsi_mod.scan=none agent.config_file=/etc/agent-config.toml
agent.enable_signature_verification=false
-pidfile
/run/vc/vm/81a554805f4d82ba05133edbba2be5099237a30527334cc33bde8d38c35359e1/pid
-smp 1,cores=1,threads=1,sockets=1,maxcpus=1
The significant portion of that output is the section after -append
but before the next option (-pidfile
in this example). Export that into an environment variable:
$ export KERNEL_APPEND="tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k cryptomgr.notests net.ifnames=0 pci=lastbus=0 console=hvc0 console=hvc1 quiet panic=1 nr_cpus=1 selinux=0 agent.aa_kbc_params=online_sev_kbc::10.8.0.194:44444 scsi_mod.scan=none agent.config_file=/etc/agent-config.toml agent.enable_signature_verification=false"
Use sev-snp-measure
to create the guest launch measurement. Then create a launch measurement verification policy by inserting the measurement into the simple-kbs's database:
$ MEASUREMENT=$(sev-snp-measure --mode=sev --output-format=base64 \
--ovmf="$FIRMWARE_PATH" --kernel="$KERNEL_PATH" \
--initrd="$INITRD_PATH" --append="${KERNEL_APPEND}")
$ echo $MEASUREMENT
tc9SbQPHZYklGwBgI0a/xh5avxeOakZ/AODXnDiWBpQ=
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} \
-e "INSERT INTO policy VALUES (10, '\[\\${MEASUREMENT}\\\]', '\[\]', 0, 0, '\[\]', now(), NULL, 1)"
$ mysql -u ${KBS_DB_USER} -p${KBS_DB_PW} -D ${KBS_DB} \
-e "UPDATE secrets SET polid = '10' WHERE secret_id = 'default/key/key_id1';"
Delete and restart the pod to apply the guest measurement policy. To get more detailed information on how simple-kbs responds to the attestation-agent requests, restart the simple-kbs with debugging messages enabled (RUST_LOG=simple_kbs
):
$ kubectl delete pod/coco-custom-nginx
pod "coco-custom-nginx" deleted
$ cd simple-kbs
$ RUST_LOG=simple_kbs ./simple-kbs --grpc_sock=0.0.0.0:44444 &
$ cd ..
$ kubectl apply -f coco-sev-nginx.yaml
pod/coco-custom-nginx created
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
coco-custom-nginx 1/1 Running 0 5m16s
You can see in simple-kbs logs that the measured digest (tc9SbQPHZYklGwBgI0a/xh5avxeOakZ/AODXnDiWBpQ=
) corresponds to expected value from the database:
[2023-05-04T13:11:15Z INFO simple_kbs::grpc] Policy validated
succesfully. Connection: Connection { policy: 3, fw_api_major: 1,
fw_api_minor: 42, fw_build_id: 42, launch_description: "shim
launch", fw_digest: "tc9SbQPHZYklGwBgI0a/xh5avxeOakZ/AODXnDiWBpQ=" }
Encrypted containers
In this article, I've demonstrated how to create Confidential Containers using EPYC™ hardware with AMD SEV. Enabling and using SEV defends the memory space utilized by the workload through hardware encryption. In addition to this defense, the launch measurement provided by SEV allows for a policy-based key release to the orchestrator scheduler for decrypting a container image. These methods showcase how the industry is moving towards cryptographic solutions to protect cloud data and infrastructure.
For more details on the CoCo project, read the What is the Confidential Containers project. To understand the topic of measurements, read Understanding the Confidential Containers Attestation Flow.
I would like to acknowledge and thank Beraldo Leal for reviewing this article, and for going through the steps to install and configure Confidential Containers on his AMD SEV machine.
Über die Autoren
Wainer has worked with build, release, CI and testing of open source projects since 2017 when he joined the Red Hat Virtualization team. His recent focus has been on turning Confidential Containers into a mature technology.
Ryan Savino has been a Software Design Engineer at AMD for the past two years. His team works on enabling the open source community in projects related to AMD security and performance.
Mehr davon
Nach Thema durchsuchen
Automatisierung
Das Neueste zum Thema IT-Automatisierung für Technologien, Teams und Umgebungen
Künstliche Intelligenz
Erfahren Sie das Neueste von den Plattformen, die es Kunden ermöglichen, KI-Workloads beliebig auszuführen
Open Hybrid Cloud
Erfahren Sie, wie wir eine flexiblere Zukunft mit Hybrid Clouds schaffen.
Sicherheit
Erfahren Sie, wie wir Risiken in verschiedenen Umgebungen und Technologien reduzieren
Edge Computing
Erfahren Sie das Neueste von den Plattformen, die die Operations am Edge vereinfachen
Infrastruktur
Erfahren Sie das Neueste von der weltweit führenden Linux-Plattform für Unternehmen
Anwendungen
Entdecken Sie unsere Lösungen für komplexe Herausforderungen bei Anwendungen
Original Shows
Interessantes von den Experten, die die Technologien in Unternehmen mitgestalten
Produkte
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud-Services
- Alle Produkte anzeigen
Tools
- Training & Zertifizierung
- Eigenes Konto
- Kundensupport
- Für Entwickler
- Partner finden
- Red Hat Ecosystem Catalog
- Mehrwert von Red Hat berechnen
- Dokumentation
Testen, kaufen und verkaufen
Kommunizieren
Über Red Hat
Als weltweit größter Anbieter von Open-Source-Software-Lösungen für Unternehmen stellen wir Linux-, Cloud-, Container- und Kubernetes-Technologien bereit. Wir bieten robuste Lösungen, die es Unternehmen erleichtern, plattform- und umgebungsübergreifend zu arbeiten – vom Rechenzentrum bis zum Netzwerkrand.
Wählen Sie eine Sprache
Red Hat legal and privacy links
- Über Red Hat
- Jobs bei Red Hat
- Veranstaltungen
- Standorte
- Red Hat kontaktieren
- Red Hat Blog
- Diversität, Gleichberechtigung und Inklusion
- Cool Stuff Store
- Red Hat Summit