Peer-pods is a new Red Hat OpenShift feature that enables an OpenShift sandboxed container (OSC) running on a bare-metal deployment to run on OpenShift in a public cloud and on VMware. It's not uncommon to want to run OpenShift in a virtual machine instead of on the bare-metal nodes. While it's possible to run a virtual machine inside a virtual machine, it demands a whole new subset of support concerns when you do it in production. In this article, I'll demonstrate how to solve this problem, using a combination of peer-pods and libvirt. By the end of this tutorial, you'll know how to create a VM image for a peer-pod deployment, and how to run a workload in a peer-pod using the host's libvirt.
Peer-pod libvirt support is still experimental. It's not officially supported by Red Hat, but I encourage you to experiment and share your feedback to improve the peer-pods solution. This article assumes you're familiar with Red Hat OpenShift and the libvirt provider.
The advantage of peer-pods
A peer-pod isn't restricted to just a cloud provider. It can be deployed on-premise in a locally installed OpenShift cluster. The architecture remains the same, together with all the benefits explained in the peer-pods solution overview blog. Once you've proven the solution is viable, you can enjoy those benefits in an on-premise OpenShift install.
Setup
To follow along with this tutorial, you need:
- git
- tar
- podman
- libvirt
- virsh
Additionally, you must have an OpenShift cluster, version 4.13 or higher. For the purpose of this guide, I assume the cluster was created using kcli. You must have this information about your cluster available:
- Network name (
LIBVIRT_NET_NAME
):kcli list network
- IP address (
LIBVIRT_NET_IP
):kcli list network
- Pool name (
LIBVIRT_POOL_NAME
):virsh -c qemu:///system pool-list
- SSH keys (
LIBVIRT_KEY_PUB
andLIBVIRT_KEY_PRIV
): This can be provided bykcli
or it can be generated using OpenSSH - Vxlan port (9000 by default)
Download the image
You need a virtual image that can run in libvirt. You'll invoke this image on the guest VM instead of the host bare-metal server. I recommend either CentOS or RHEL.
CentOS image
$ export IMAGE_PROVIDER=centos
$ export ARCH=amd64
$ curl \
https://raw.githubusercontent.com/confidential-containers/cloud-api-adaptor/v0.5.0/podvm/hack/download-image.sh | bash -s \
quay.io/confidential-containers/podvm-generic-$IMAGE_PROVIDER-$ARCH . \
-o podvm-amd64.qcow2'*.tar'
This creates an image file called podvm-amd64.qcow2
.
RHEL image
For a RHEL 8 or 9 image, follow the steps shown in peer-pods solution overview blog. At the end of the process, you have a qcow2 image. For the purpose of this guide, name the image podvm-amd64.qcow2
.
Set default image
Next, set the default image:
$ export IMAGE=path/to/podvm-amd64.qcow2
$ virsh -c qemu:///system vol-create-as --pool default \
--name podvm-base.qcow2 --capacity 20G \
--allocation 2G --prealloc-metadata --format qcow2
$ virsh -c qemu:///system vol-upload --vol podvm-base.qcow2 \
$IMAGE --pool default --sparse
Should this step fail because podvm-base.qcow2
already exists, topy the existing podvm-base.qcow2
file and rename it:
$ NEW_PODVM_NAME=podvm-base-centos.qcow2
$ virsh -c qemu:///system vol-clone \
podvm-base.qcow2 $NEW_PODVM_NAME --pool default
Then delete podvm-base.qcow2
:
$ virsh -c qemu:///system vol-delete podvm-base.qcow2 --pool default
And then recreate podvm-base.qcow2
as demonstrated previously.
Verify that the image has been set:
$ virsh -c qemu:///system vol-info --pool default podvm-base.qcow2
Name: podvm-base.qcow2
Type: file
Capacity: 6.00 GiB
Allocation: 631.52 MiB
Deploy the OSC operator
To deplay the 1.4.0 operator, go to the marketplace and install it. For step-by-step instructions, read the official installation guide.
The libvirt provider is not currently supported officially in OSC 1.4.0, so you must update the internal coud-api-adaptor (CAA) to an upstream version:
$ export UPSTREAM_CAA=quay.io/confidential-containers/cloud-api-adaptor:latest
$ oc set image ds/peerpodconfig-ctrl-caa-daemon \
-n openshift-sandboxed-containers-operator \
cc-runtime-install-pod=$UPSTREAM_CAA
Collect peer-pods configuration parameters for libvirt
Now you must prepare the operator to run with peer-pods.
- First, create and deploy a peer-pods Secret YAML definition:
cat > peer-pods-secret.yaml <<EOF
apiVersion: v1
kind: Secret
metadata:
name: peer-pods-secret
namespace: openshift-sandboxed-containers-operator
type: Opaque
stringData:
CLOUD_PROVIDER: "libvirt"
VXLAN_PORT: "${LIBVIRT_VXLAN_PORT}"
LIBVIRT_URI: "qemu+ssh://root@${LIBVIRT_NET_IP}.1/system?no_verify=1"
LIBVIRT_NET: "${LIBVIRT_NET_NAME}"
LIBVIRT_POOL: "${LIBVIRT_POOL_NAME}"
EOF
$ oc apply -f peer-pods-secret.yaml
- Create and deploy the peer-pods ConfigMap YAML definition:
$ cat > peer-pods-config-map.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: peer-pods-cm
namespace: openshift-sandboxed-containers-operator
data:
CLOUD_PROVIDER: "libvirt"
EOF
$ oc apply -f peer-pods-config-map.yaml
- You need an SSH key. The Kubernetes Secret contains the private key, which the CAA pod uses to connect to the host's libvirt, which in turn has the public key in
/root/.ssh/authorized_keys
.
$ oc create secret generic ssh-key-secret \
-n openshift-sandboxed-containers-operator \
--from-file=id_rsa.pub=${LIBVIRT_KEY_PUB} \
--from-file=id_rsa=${LIBVIRT_KEY_PRIV}
Alternatively, if no is LIBVIRT_KEY_*
is present, you can generate one and add it to /root/.ssh/authorized_keys
:
$ ssh-keygen -f ./id_rsa -N ""
$ oc create secret generic ssh-key-secret \
-n openshift-sandboxed-containers-operator \
--from-file=id_rsa.pub=./id_rsa.pub \
--from-file=id_rsa=./id_rsa
If you need to modify any of the parameters in the peer-pods-cm
and peer-pods-secret
objects, you can change values in the respective YAML files, and then deploy the files using oc apply -f example.yaml
. Restart the cloud-api-adaptor
pods with:
$ oc set env ds/peerpodconfig-ctrl-caa-daemon \
-n openshift-sandboxed-containers-operator \
REBOOT="$(date)"
Set up the peer-pods
You're ready to run peer-pods.
First, create KataConfig CRD from the Openshift Sandboxed Containers operator page, or deploy KataConfig from the terminal:
$ cat > kataconfig.yaml <<EOF
apiVersion: kataconfiguration.openshift.io/v1
kind: KataConfig
metadata:
name: example-kataconfig
spec:
enablePeerPods: true
EOF
$ oc apply -f kataconfig.yaml
Wait for installation to complete:
$ oc wait --for=condition=Updating=false \
machineconfigpool/kata-oc --timeout=-1s
$ oc rollout status daemonset peerpodconfig-ctrl-caa-daemon \
-n openshift-sandboxed-containers-operator --timeout=60s
If you're using the web console, switch to the openshift-sandboxed-containers-operator namespace and navigate to Workloads > Pods. Depending on your cluster setup, there are at least controller-manager-{} and peerpodconfig-ctrl-caa-daemon-{} pods in the Running state.
Next, you must update the coud-api-adaptor (CAA). Wait until the peerpodconfig-ctrl-caa-daemon-{*}
objects are created. They're likely to show up in the Error/CrashBackLoop
state, because the default operator doesn't accept libvirt as a cloud provider. To solve this issue, update the CAA to use the upstream code:
$ UPSTREAM_CAA=quay.io/confidential-containers/cloud-api-adaptor:latest
$ oc set image ds/peerpodconfig-ctrl-caa-daemon \
-n openshift-sandboxed-containers-operator \
cc-runtime-install-pod=$UPSTREAM_CAA
After that, the pods automatically restart. This can take some time, so be patient and monitor your cluster using your browser or the oc wait
command.
After the installation's complete, you have two runtime classes:
- kata: Creates sandboxed containers using Qemu and KVM. This one is created by default.
- kata-remote-cc: Creates sandboxed containers using peer-pods.
Peer-pods has been installed!
Run a peer-pod workload
Now that you've installed peer-pods, you can run your first pod. The example hello-openshift
web application prints "Hello OpenShift!" to visitors. Try it out.
- Create a
hello-openshift.yaml
file with the following contents:
apiVersion: v1 kind: Pod metadata: name: hello-openshift labels: app: hello-openshift spec: runtimeClassName: kata-remote-cc containers: - name: hello-openshift image: quay.io/openshift/origin-hello-openshift ports: - containerPort: 8888 securityContext: privileged: false allowPrivilegeEscalation: false runAsNonRoot: true runAsUser: 1001 capabilities: drop: - ALL seccompProfile: type: RuntimeDefault --- kind: Service apiVersion: v1 metadata: name: hello-openshift-service labels: app: hello-openshift spec: selector: app: hello-openshift ports: - port: 8888
- Deploy it:
$ oc apply -f hello-openshift.yaml
$ oc get pod/hello-openshift
- Create an OpenShift route:
$ oc expose service hello-openshift-service -l app=hello-openshift
$ export APP_URL=$(oc get routes/hello-openshift-service -o jsonpath='{.spec.host}')
Done! Now verify that the application is responding:
$ curl ${APP_URL}
Inspecting peer-pods
To learn how to inspect peer-pods to gain insight into how and what they're running, read this article.
Undeploy peer-pods
To undeploy your peer-pods:
- Delete
hello-openshift
:
$ oc delete all -l app=hello-openshift
- Delete
kataconfig
:
$ oc delete kataconfigs/example-kataconfig \
-n openshift-sandboxed-containers-operator
If this command hangs, do NOT cancel the command. In a separate terminal, run oc edit kataconfigs/example-kataconfig
and remove the section called "finalizers".
- Delete
ConfigMap
:
$ oc delete configmaps/peer-pods-cm \
-n openshift-sandboxed-containers-operator
- Delete
PeerPodsSecret
:
$ oc delete secrets/peer-pods-secret \
-n openshift-sandboxed-containers-operator
- Delete the operator from the Openshift Marketplace.
Conclusion
You've installed and deployed peer-pods, and tested a simple example web application. For further information, refer to part one and part two of this series.
執筆者紹介
Emanuele Giuseppe Esposito is a Software Engineer at Red Hat, with focus on Confidential Computing, QEMU and KVM. He joined Red Hat in 2021, right after getting a Master Degree in CS at ETH Zürich. Emanuele is passionate about the whole virtualization stack, ranging from Openshift Sandboxed Containers to low-level features in QEMU and KVM.
類似検索
チャンネル別に見る
自動化
テクノロジー、チームおよび環境に関する IT 自動化の最新情報
AI (人工知能)
お客様が AI ワークロードをどこでも自由に実行することを可能にするプラットフォームについてのアップデート
オープン・ハイブリッドクラウド
ハイブリッドクラウドで柔軟に未来を築く方法をご確認ください。
セキュリティ
環境やテクノロジー全体に及ぶリスクを軽減する方法に関する最新情報
エッジコンピューティング
エッジでの運用を単純化するプラットフォームのアップデート
インフラストラクチャ
世界有数のエンタープライズ向け Linux プラットフォームの最新情報
アプリケーション
アプリケーションの最も困難な課題に対する Red Hat ソリューションの詳細
オリジナル番組
エンタープライズ向けテクノロジーのメーカーやリーダーによるストーリー
製品
ツール
試用、購入、販売
コミュニケーション
Red Hat について
エンタープライズ・オープンソース・ソリューションのプロバイダーとして世界をリードする Red Hat は、Linux、クラウド、コンテナ、Kubernetes などのテクノロジーを提供しています。Red Hat は強化されたソリューションを提供し、コアデータセンターからネットワークエッジまで、企業が複数のプラットフォームおよび環境間で容易に運用できるようにしています。
言語を選択してください
Red Hat legal and privacy links
- Red Hat について
- 採用情報
- イベント
- 各国のオフィス
- Red Hat へのお問い合わせ
- Red Hat ブログ
- ダイバーシティ、エクイティ、およびインクルージョン
- Cool Stuff Store
- Red Hat Summit