Cloud-native architectures and containers can increase efficiency, performance, and agility for telecommunications service providers as they continue to find ways to compete in the market. Developers, architects, and others involved in the software development process can consider a shift-on-stack deployment model with vhostuser to integrate cloud-native network functions (CNFs).
In network functions virtualization (NFV) deployments, OpenShift can be deployed on top of OpenStack, but the deployment only supports exposing SR-IOV Virtual Functions to the CNF running on the OpenShift worker node (VM running on OpenStack).
As many NFV deployments support userspace fast datapath via OvS-DPDK, it would be advantageous for CNFs to utilize the OvS-DPDK FDP in addition to SR-IOV. In this post, we will demonstrate the steps required to deploy a CNF with additional OvS-DPDK's vhostuser ports.
In summary, the steps we will walk through include:
-
Creating and deploying a MachineSet specifying additional fast datapath networks. The MachineSet adds fast datapath worker nodes to an existing OpenShift cluster.
-
Binding the vfio-pci driver to additional fast datapath networks using a MachineConfig specification.
-
Creating additional networks in the cluster using a NetworkAttachmentDefinition and bind those networks using host-devices.
Requisites
Before we get started with the deployment, we need to check for these system and network requirements:
-
Red Hat OpenStack 16.2 deployed with bare-metal compute nodes having OvS-DPDK FastData path enabled.
-
Red Hat OpenShift 4.9 (or 4.10) deployed on the deployed Red Hat OpenStack environment. The SR-IOV network operator is not required for using vhostuser ports.
-
performance-addon-operator deployed with CPU isolation and huge pages, which should be applied to the worker node profile.
-
Additional OvS-DPDK neutron networks created with the same configuration as other VNF applications running in the Red Hat OpenStack layer. For the following example, let's assume two VLAN-based OvS-DPDK networks are created with names "vhostuser1" and "vhostuser2". Follow the OSP NFV document for detailed steps.
Create Red Hat OpenShift Container Platform Worker Nodes with Vhostuser
You can create the OpenShift Container Platform worker nodes by following the MachineSet approach detailed in these docs. The MachineSet method allows customization of the worker node machine profile by specifying the additional network connections for a worker node in the MachineSet specification.
An Ansible role has been created to facilitate the creation of the MachineSet configuration. Follow the README in the role to create inventory.yml, which contains all the required information to create a MachineSet with additional networks.
all: hosts: localhost: ansible_connection: local ansible_python_interpreter: "{{ansible_playbook_python}}" number_of_replicas: 1 cluster_metadata_path: "~/sos-fdp/build-artifacts/metadata.json" openstack_cloud: "openstack" nova_flavor: "ocp-worker" node_role: "worker" additional_networks: - name: "vhostuser1" name_suffix: "vhostuser1" tags: - uplink_input vnic_type: "normal" driver: "vfio-pci" - name: "vhostuser2" name_suffix: "vhostuser2" tags: - uplink_output vnic_type: "normal" driver: "vfio-pci"
Next, create the MachineSet via the role and deploy it. It will create the worker nodes with the additional vhostuser ports. The primary Kubernetes network port will be created using the default OpenStack network type.
$ git clone git@github.com:rh-nfv-int/shift-on-stack-machineset.git
$ cd shift-on-stack-machineset
# create inventory.yaml as show above
$ ansible-playbook play.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml
$ oc apply -f ./build/<infraID>-worker-machineset.yaml
Once the MachineSet is applied, worker nodes will be created. Wait for the worker node to be ready before proceeding to the next step.Userspace Driver Binding on OpenShift Container Platform Worker nodes
CNFs that do packet processing on the datapath require userspace drivers to be bound to the interface inside the worker node VM. For DPDK-based CNF applications, the userspace driver vfio-pci should be bound to the interfaces so that the application can perform userspace packet processing.
OCP Worker nodes running on Red Hat CoreOS support vfio-pci driver to be used as the userspace driver. Additional configuration is required on the worker node to enable this support. Note that this is in Tech Preview and not currently supported in production. The vfio module on the created worker node should be loaded with additional flags to enable unsafe noiommu mode enable_unsafe_noiommu_mode=1. Here, MachineConfig can be created in order to create the modprobe conf file to load vfio with this option enabled.
kind: MachineConfig apiVersion: machineconfiguration.openshift.io/v1 metadata: name: 99-vfio-noiommu labels: machineconfiguration.openshift.io/role: worker spec: osImageURL: '' config: ignition: version: 2.2.0 storage: files: - filesystem: root path: "/etc/modprobe.d/vfio-noiommu.conf" contents: source: data:text/plain;charset=utf-8;base64,b3B0aW9ucyB2ZmlvIGVuYWJsZV91bnNhZmVfbm9pb21tdV9tb2RlPTEK verification: {} mode: 0644
In order to bind the vhostuser interfaces to the vfio-pci driver, a set of scripts has been added to the Ansible role shift-on-stack-machineset, which creates MachineConfig resource with these scripts files. Once the MachineConfig is applied on the worker nodes, the vfio-pci driver will be bound to the interface on every reboot. The MachineConfig creation is part of the gen.yaml playbook, which will be executed in the next step.
Create Vhostuser OpenShift Container Platform Network
Additional networks can be attached to the CNF application using the NetworkAttachmentDefintion resource. The NetworkAttachmentDefinition utilizes the host-device CNI plugin to attach vhostuser-based additional interfaces to the CNF application POD. The next section provides a sample definition of the NetworkAttachmentDefintion resource created with a specific PCI address.
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vhostuser1 namespace: example-cnf spec: config: '{ "cniVersion": "0.3.1", "name": "hostonly", "type": "host-device", "pciBusId": "0000:00:04.0", "ipam": { } }'
$ cd shift-on-stack-machineset $ ansible-playbook gen.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml $ oc apply -f build/<infraID>-<net>-netattach.yaml $ oc apply -f build/<infraID>-machine-config.yaml
The NetworkAttachmentDefinition resources for vhostuser1 and vhostuser2 networks will be created on the targeted namespace example-cnf.
$ oc -n example-cnf get net-attach-def NAME AGE vhostuser1 5h44m vhostuser2 5h44m
Create CNF Application with Vhostuser
Finally, let’s create a CNF application using testpmd by adding the created networks by using this template format.
apiVersion: v1 kind: Pod metadata: name: testpmd namespace: example-cnf annotations: k8s.v1.cni.cncf.io/networks: "vhostuser1,vhostuser2" spec: containers: - name: testpmd command: ["/bin/sh"] args: ["-c", "testpmd -l $(taskset -pc 1 | cut -d: -f2) --in-memory -w 00:04.0 -w 00:05.0 --socket-mem 1024 -n 4 -- --nb-cores=1 --auto-start --forward-mode=mac --stats-period 10"] image: registry.redhat.io/openshift4/dpdk-base-rhel8:v4.6 securityContext: privileged: true runAsUser: 0 resources: requests: memory: 1000Mi hugepages-1Gi: 3Gi cpu: '3' limits: hugepages-1Gi: 3Gi cpu: '3' memory: 1000Mi volumeMounts: - mountPath: /dev/hugepages name: hugepage readOnly: False volumes: - name: hugepage emptyDir: medium: HugePages
NOTE: For shift-on-stack CNF applications, the security context "privileged: true" is mandatory to allow the application to access PCI devices. This is a limitation of the current version.
Conclusion
As many of the network function vendors are transitioning from the VNF to CNF era, the shift-on-stack deployment model with vhostuser provides a head start in creating a NFV-ready environment to develop and integrate CNFs.
This deployment strategy gives an opportunity for service providers to experience CNFs on the existing Red Hat OpenStack Platform.
Learn more about our range of solutions that can help telco providers with flexible deployment configurations in this blog post.
Über die Autoren
Saravanan KR is a Principal Software Engineer at Red Hat.
Emilien is a French citizen living in Canada (QC) who has been contributing to OpenStack since 2011 when it was still a young project. While his major focus has been the installer, his impact has helped customers have a better experience when deploying, upgrading and operating OpenStack at large scale. Technical and team leader at Red Hat, he's developing leadership skills with passion for teamwork and technical challenges. He loves sharing his knowledge and often give talks to conferences.
Aaron Smith is a Senior Principal Software Engineer at Red Hat.
Nach Thema durchsuchen
Automatisierung
Das Neueste zum Thema IT-Automatisierung für Technologien, Teams und Umgebungen
Künstliche Intelligenz
Erfahren Sie das Neueste von den Plattformen, die es Kunden ermöglichen, KI-Workloads beliebig auszuführen
Open Hybrid Cloud
Erfahren Sie, wie wir eine flexiblere Zukunft mit Hybrid Clouds schaffen.
Sicherheit
Erfahren Sie, wie wir Risiken in verschiedenen Umgebungen und Technologien reduzieren
Edge Computing
Erfahren Sie das Neueste von den Plattformen, die die Operations am Edge vereinfachen
Infrastruktur
Erfahren Sie das Neueste von der weltweit führenden Linux-Plattform für Unternehmen
Anwendungen
Entdecken Sie unsere Lösungen für komplexe Herausforderungen bei Anwendungen
Original Shows
Interessantes von den Experten, die die Technologien in Unternehmen mitgestalten
Produkte
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud-Services
- Alle Produkte anzeigen
Tools
- Training & Zertifizierung
- Eigenes Konto
- Kundensupport
- Für Entwickler
- Partner finden
- Red Hat Ecosystem Catalog
- Mehrwert von Red Hat berechnen
- Dokumentation
Testen, kaufen und verkaufen
Kommunizieren
Über Red Hat
Als weltweit größter Anbieter von Open-Source-Software-Lösungen für Unternehmen stellen wir Linux-, Cloud-, Container- und Kubernetes-Technologien bereit. Wir bieten robuste Lösungen, die es Unternehmen erleichtern, plattform- und umgebungsübergreifend zu arbeiten – vom Rechenzentrum bis zum Netzwerkrand.
Wählen Sie eine Sprache
Red Hat legal and privacy links
- Über Red Hat
- Jobs bei Red Hat
- Veranstaltungen
- Standorte
- Red Hat kontaktieren
- Red Hat Blog
- Diversität, Gleichberechtigung und Inklusion
- Cool Stuff Store
- Red Hat Summit