Cloud-native architectures and containers can increase efficiency, performance, and agility for telecommunications service providers as they continue to find ways to compete in the market. Developers, architects, and others involved in the software development process can consider a shift-on-stack deployment model with vhostuser to integrate cloud-native network functions (CNFs).
In network functions virtualization (NFV) deployments, OpenShift can be deployed on top of OpenStack, but the deployment only supports exposing SR-IOV Virtual Functions to the CNF running on the OpenShift worker node (VM running on OpenStack).
As many NFV deployments support userspace fast datapath via OvS-DPDK, it would be advantageous for CNFs to utilize the OvS-DPDK FDP in addition to SR-IOV. In this post, we will demonstrate the steps required to deploy a CNF with additional OvS-DPDK's vhostuser ports.
In summary, the steps we will walk through include:
-
Creating and deploying a MachineSet specifying additional fast datapath networks. The MachineSet adds fast datapath worker nodes to an existing OpenShift cluster.
-
Binding the vfio-pci driver to additional fast datapath networks using a MachineConfig specification.
-
Creating additional networks in the cluster using a NetworkAttachmentDefinition and bind those networks using host-devices.
Requisites
Before we get started with the deployment, we need to check for these system and network requirements:
-
Red Hat OpenStack 16.2 deployed with bare-metal compute nodes having OvS-DPDK FastData path enabled.
-
Red Hat OpenShift 4.9 (or 4.10) deployed on the deployed Red Hat OpenStack environment. The SR-IOV network operator is not required for using vhostuser ports.
-
performance-addon-operator deployed with CPU isolation and huge pages, which should be applied to the worker node profile.
-
Additional OvS-DPDK neutron networks created with the same configuration as other VNF applications running in the Red Hat OpenStack layer. For the following example, let's assume two VLAN-based OvS-DPDK networks are created with names "vhostuser1" and "vhostuser2". Follow the OSP NFV document for detailed steps.
Create Red Hat OpenShift Container Platform Worker Nodes with Vhostuser
You can create the OpenShift Container Platform worker nodes by following the MachineSet approach detailed in these docs. The MachineSet method allows customization of the worker node machine profile by specifying the additional network connections for a worker node in the MachineSet specification.
An Ansible role has been created to facilitate the creation of the MachineSet configuration. Follow the README in the role to create inventory.yml, which contains all the required information to create a MachineSet with additional networks.
all: hosts: localhost: ansible_connection: local ansible_python_interpreter: "{{ansible_playbook_python}}" number_of_replicas: 1 cluster_metadata_path: "~/sos-fdp/build-artifacts/metadata.json" openstack_cloud: "openstack" nova_flavor: "ocp-worker" node_role: "worker" additional_networks: - name: "vhostuser1" name_suffix: "vhostuser1" tags: - uplink_input vnic_type: "normal" driver: "vfio-pci" - name: "vhostuser2" name_suffix: "vhostuser2" tags: - uplink_output vnic_type: "normal" driver: "vfio-pci"
Next, create the MachineSet via the role and deploy it. It will create the worker nodes with the additional vhostuser ports. The primary Kubernetes network port will be created using the default OpenStack network type.
$ git clone git@github.com:rh-nfv-int/shift-on-stack-machineset.git
$ cd shift-on-stack-machineset
# create inventory.yaml as show above
$ ansible-playbook play.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml
$ oc apply -f ./build/<infraID>-worker-machineset.yaml
Once the MachineSet is applied, worker nodes will be created. Wait for the worker node to be ready before proceeding to the next step.Userspace Driver Binding on OpenShift Container Platform Worker nodes
CNFs that do packet processing on the datapath require userspace drivers to be bound to the interface inside the worker node VM. For DPDK-based CNF applications, the userspace driver vfio-pci should be bound to the interfaces so that the application can perform userspace packet processing.
OCP Worker nodes running on Red Hat CoreOS support vfio-pci driver to be used as the userspace driver. Additional configuration is required on the worker node to enable this support. Note that this is in Tech Preview and not currently supported in production. The vfio module on the created worker node should be loaded with additional flags to enable unsafe noiommu mode enable_unsafe_noiommu_mode=1. Here, MachineConfig can be created in order to create the modprobe conf file to load vfio with this option enabled.
kind: MachineConfig apiVersion: machineconfiguration.openshift.io/v1 metadata: name: 99-vfio-noiommu labels: machineconfiguration.openshift.io/role: worker spec: osImageURL: '' config: ignition: version: 2.2.0 storage: files: - filesystem: root path: "/etc/modprobe.d/vfio-noiommu.conf" contents: source: data:text/plain;charset=utf-8;base64,b3B0aW9ucyB2ZmlvIGVuYWJsZV91bnNhZmVfbm9pb21tdV9tb2RlPTEK verification: {} mode: 0644
In order to bind the vhostuser interfaces to the vfio-pci driver, a set of scripts has been added to the Ansible role shift-on-stack-machineset, which creates MachineConfig resource with these scripts files. Once the MachineConfig is applied on the worker nodes, the vfio-pci driver will be bound to the interface on every reboot. The MachineConfig creation is part of the gen.yaml playbook, which will be executed in the next step.
Create Vhostuser OpenShift Container Platform Network
Additional networks can be attached to the CNF application using the NetworkAttachmentDefintion resource. The NetworkAttachmentDefinition utilizes the host-device CNI plugin to attach vhostuser-based additional interfaces to the CNF application POD. The next section provides a sample definition of the NetworkAttachmentDefintion resource created with a specific PCI address.
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vhostuser1 namespace: example-cnf spec: config: '{ "cniVersion": "0.3.1", "name": "hostonly", "type": "host-device", "pciBusId": "0000:00:04.0", "ipam": { } }'
$ cd shift-on-stack-machineset $ ansible-playbook gen.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml $ oc apply -f build/<infraID>-<net>-netattach.yaml $ oc apply -f build/<infraID>-machine-config.yaml
The NetworkAttachmentDefinition resources for vhostuser1 and vhostuser2 networks will be created on the targeted namespace example-cnf.
$ oc -n example-cnf get net-attach-def NAME AGE vhostuser1 5h44m vhostuser2 5h44m
Create CNF Application with Vhostuser
Finally, let’s create a CNF application using testpmd by adding the created networks by using this template format.
apiVersion: v1 kind: Pod metadata: name: testpmd namespace: example-cnf annotations: k8s.v1.cni.cncf.io/networks: "vhostuser1,vhostuser2" spec: containers: - name: testpmd command: ["/bin/sh"] args: ["-c", "testpmd -l $(taskset -pc 1 | cut -d: -f2) --in-memory -w 00:04.0 -w 00:05.0 --socket-mem 1024 -n 4 -- --nb-cores=1 --auto-start --forward-mode=mac --stats-period 10"] image: registry.redhat.io/openshift4/dpdk-base-rhel8:v4.6 securityContext: privileged: true runAsUser: 0 resources: requests: memory: 1000Mi hugepages-1Gi: 3Gi cpu: '3' limits: hugepages-1Gi: 3Gi cpu: '3' memory: 1000Mi volumeMounts: - mountPath: /dev/hugepages name: hugepage readOnly: False volumes: - name: hugepage emptyDir: medium: HugePages
NOTE: For shift-on-stack CNF applications, the security context "privileged: true" is mandatory to allow the application to access PCI devices. This is a limitation of the current version.
Conclusion
As many of the network function vendors are transitioning from the VNF to CNF era, the shift-on-stack deployment model with vhostuser provides a head start in creating a NFV-ready environment to develop and integrate CNFs.
This deployment strategy gives an opportunity for service providers to experience CNFs on the existing Red Hat OpenStack Platform.
Learn more about our range of solutions that can help telco providers with flexible deployment configurations in this blog post.
저자 소개
Saravanan KR is a Principal Software Engineer at Red Hat.
Emilien is a French citizen living in Canada (QC) who has been contributing to OpenStack since 2011 when it was still a young project. While his major focus has been the installer, his impact has helped customers have a better experience when deploying, upgrading and operating OpenStack at large scale. Technical and team leader at Red Hat, he's developing leadership skills with passion for teamwork and technical challenges. He loves sharing his knowledge and often give talks to conferences.
Aaron Smith is a Senior Principal Software Engineer at Red Hat.
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.