Cloud-native architectures and containers can increase efficiency, performance, and agility for telecommunications service providers as they continue to find ways to compete in the market. Developers, architects, and others involved in the software development process can consider a shift-on-stack deployment model with vhostuser to integrate cloud-native network functions (CNFs).
In network functions virtualization (NFV) deployments, OpenShift can be deployed on top of OpenStack, but the deployment only supports exposing SR-IOV Virtual Functions to the CNF running on the OpenShift worker node (VM running on OpenStack).
As many NFV deployments support userspace fast datapath via OvS-DPDK, it would be advantageous for CNFs to utilize the OvS-DPDK FDP in addition to SR-IOV. In this post, we will demonstrate the steps required to deploy a CNF with additional OvS-DPDK's vhostuser ports.
In summary, the steps we will walk through include:
-
Creating and deploying a MachineSet specifying additional fast datapath networks. The MachineSet adds fast datapath worker nodes to an existing OpenShift cluster.
-
Binding the vfio-pci driver to additional fast datapath networks using a MachineConfig specification.
-
Creating additional networks in the cluster using a NetworkAttachmentDefinition and bind those networks using host-devices.
Requisites
Before we get started with the deployment, we need to check for these system and network requirements:
-
Red Hat OpenStack 16.2 deployed with bare-metal compute nodes having OvS-DPDK FastData path enabled.
-
Red Hat OpenShift 4.9 (or 4.10) deployed on the deployed Red Hat OpenStack environment. The SR-IOV network operator is not required for using vhostuser ports.
-
performance-addon-operator deployed with CPU isolation and huge pages, which should be applied to the worker node profile.
-
Additional OvS-DPDK neutron networks created with the same configuration as other VNF applications running in the Red Hat OpenStack layer. For the following example, let's assume two VLAN-based OvS-DPDK networks are created with names "vhostuser1" and "vhostuser2". Follow the OSP NFV document for detailed steps.
Create Red Hat OpenShift Container Platform Worker Nodes with Vhostuser
You can create the OpenShift Container Platform worker nodes by following the MachineSet approach detailed in these docs. The MachineSet method allows customization of the worker node machine profile by specifying the additional network connections for a worker node in the MachineSet specification.
An Ansible role has been created to facilitate the creation of the MachineSet configuration. Follow the README in the role to create inventory.yml, which contains all the required information to create a MachineSet with additional networks.
all: hosts: localhost: ansible_connection: local ansible_python_interpreter: "{{ansible_playbook_python}}" number_of_replicas: 1 cluster_metadata_path: "~/sos-fdp/build-artifacts/metadata.json" openstack_cloud: "openstack" nova_flavor: "ocp-worker" node_role: "worker" additional_networks: - name: "vhostuser1" name_suffix: "vhostuser1" tags: - uplink_input vnic_type: "normal" driver: "vfio-pci" - name: "vhostuser2" name_suffix: "vhostuser2" tags: - uplink_output vnic_type: "normal" driver: "vfio-pci"
Next, create the MachineSet via the role and deploy it. It will create the worker nodes with the additional vhostuser ports. The primary Kubernetes network port will be created using the default OpenStack network type.
$ git clone git@github.com:rh-nfv-int/shift-on-stack-machineset.git
$ cd shift-on-stack-machineset
# create inventory.yaml as show above
$ ansible-playbook play.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml
$ oc apply -f ./build/<infraID>-worker-machineset.yaml
Once the MachineSet is applied, worker nodes will be created. Wait for the worker node to be ready before proceeding to the next step.Userspace Driver Binding on OpenShift Container Platform Worker nodes
CNFs that do packet processing on the datapath require userspace drivers to be bound to the interface inside the worker node VM. For DPDK-based CNF applications, the userspace driver vfio-pci should be bound to the interfaces so that the application can perform userspace packet processing.
OCP Worker nodes running on Red Hat CoreOS support vfio-pci driver to be used as the userspace driver. Additional configuration is required on the worker node to enable this support. Note that this is in Tech Preview and not currently supported in production. The vfio module on the created worker node should be loaded with additional flags to enable unsafe noiommu mode enable_unsafe_noiommu_mode=1. Here, MachineConfig can be created in order to create the modprobe conf file to load vfio with this option enabled.
kind: MachineConfig apiVersion: machineconfiguration.openshift.io/v1 metadata: name: 99-vfio-noiommu labels: machineconfiguration.openshift.io/role: worker spec: osImageURL: '' config: ignition: version: 2.2.0 storage: files: - filesystem: root path: "/etc/modprobe.d/vfio-noiommu.conf" contents: source: data:text/plain;charset=utf-8;base64,b3B0aW9ucyB2ZmlvIGVuYWJsZV91bnNhZmVfbm9pb21tdV9tb2RlPTEK verification: {} mode: 0644
In order to bind the vhostuser interfaces to the vfio-pci driver, a set of scripts has been added to the Ansible role shift-on-stack-machineset, which creates MachineConfig resource with these scripts files. Once the MachineConfig is applied on the worker nodes, the vfio-pci driver will be bound to the interface on every reboot. The MachineConfig creation is part of the gen.yaml playbook, which will be executed in the next step.
Create Vhostuser OpenShift Container Platform Network
Additional networks can be attached to the CNF application using the NetworkAttachmentDefintion resource. The NetworkAttachmentDefinition utilizes the host-device CNI plugin to attach vhostuser-based additional interfaces to the CNF application POD. The next section provides a sample definition of the NetworkAttachmentDefintion resource created with a specific PCI address.
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vhostuser1 namespace: example-cnf spec: config: '{ "cniVersion": "0.3.1", "name": "hostonly", "type": "host-device", "pciBusId": "0000:00:04.0", "ipam": { } }'
$ cd shift-on-stack-machineset $ ansible-playbook gen.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml $ oc apply -f build/<infraID>-<net>-netattach.yaml $ oc apply -f build/<infraID>-machine-config.yaml
The NetworkAttachmentDefinition resources for vhostuser1 and vhostuser2 networks will be created on the targeted namespace example-cnf.
$ oc -n example-cnf get net-attach-def NAME AGE vhostuser1 5h44m vhostuser2 5h44m
Create CNF Application with Vhostuser
Finally, let’s create a CNF application using testpmd by adding the created networks by using this template format.
apiVersion: v1 kind: Pod metadata: name: testpmd namespace: example-cnf annotations: k8s.v1.cni.cncf.io/networks: "vhostuser1,vhostuser2" spec: containers: - name: testpmd command: ["/bin/sh"] args: ["-c", "testpmd -l $(taskset -pc 1 | cut -d: -f2) --in-memory -w 00:04.0 -w 00:05.0 --socket-mem 1024 -n 4 -- --nb-cores=1 --auto-start --forward-mode=mac --stats-period 10"] image: registry.redhat.io/openshift4/dpdk-base-rhel8:v4.6 securityContext: privileged: true runAsUser: 0 resources: requests: memory: 1000Mi hugepages-1Gi: 3Gi cpu: '3' limits: hugepages-1Gi: 3Gi cpu: '3' memory: 1000Mi volumeMounts: - mountPath: /dev/hugepages name: hugepage readOnly: False volumes: - name: hugepage emptyDir: medium: HugePages
NOTE: For shift-on-stack CNF applications, the security context "privileged: true" is mandatory to allow the application to access PCI devices. This is a limitation of the current version.
Conclusion
As many of the network function vendors are transitioning from the VNF to CNF era, the shift-on-stack deployment model with vhostuser provides a head start in creating a NFV-ready environment to develop and integrate CNFs.
This deployment strategy gives an opportunity for service providers to experience CNFs on the existing Red Hat OpenStack Platform.
Learn more about our range of solutions that can help telco providers with flexible deployment configurations in this blog post.
執筆者紹介
Saravanan KR is a Principal Software Engineer at Red Hat.
Emilien is a French citizen living in Canada (QC) who has been contributing to OpenStack since 2011 when it was still a young project. While his major focus has been the installer, his impact has helped customers have a better experience when deploying, upgrading and operating OpenStack at large scale. Technical and team leader at Red Hat, he's developing leadership skills with passion for teamwork and technical challenges. He loves sharing his knowledge and often give talks to conferences.
Aaron Smith is a Senior Principal Software Engineer at Red Hat.
チャンネル別に見る
自動化
テクノロジー、チームおよび環境に関する IT 自動化の最新情報
AI (人工知能)
お客様が AI ワークロードをどこでも自由に実行することを可能にするプラットフォームについてのアップデート
オープン・ハイブリッドクラウド
ハイブリッドクラウドで柔軟に未来を築く方法をご確認ください。
セキュリティ
環境やテクノロジー全体に及ぶリスクを軽減する方法に関する最新情報
エッジコンピューティング
エッジでの運用を単純化するプラットフォームのアップデート
インフラストラクチャ
世界有数のエンタープライズ向け Linux プラットフォームの最新情報
アプリケーション
アプリケーションの最も困難な課題に対する Red Hat ソリューションの詳細
オリジナル番組
エンタープライズ向けテクノロジーのメーカーやリーダーによるストーリー
製品
ツール
試用、購入、販売
コミュニケーション
Red Hat について
エンタープライズ・オープンソース・ソリューションのプロバイダーとして世界をリードする Red Hat は、Linux、クラウド、コンテナ、Kubernetes などのテクノロジーを提供しています。Red Hat は強化されたソリューションを提供し、コアデータセンターからネットワークエッジまで、企業が複数のプラットフォームおよび環境間で容易に運用できるようにしています。
言語を選択してください
Red Hat legal and privacy links
- Red Hat について
- 採用情報
- イベント
- 各国のオフィス
- Red Hat へのお問い合わせ
- Red Hat ブログ
- ダイバーシティ、エクイティ、およびインクルージョン
- Cool Stuff Store
- Red Hat Summit