Shift-on-Stack with Vhostuser Ports

Cloud-native architectures and containers can increase efficiency, performance, and agility for telecommunications service providers as they continue to find ways to compete in the market. Developers, architects, and others involved in the software development process can consider a shift-on-stack deployment model with vhostuser to integrate cloud-native network functions (CNFs).

In network functions virtualization (NFV) deployments, OpenShift can be deployed on top of OpenStack, but the deployment only supports exposing SR-IOV Virtual Functions to the CNF running on the OpenShift worker node (VM running on OpenStack). 

As many NFV deployments support userspace fast datapath via OvS-DPDK, it would be advantageous for CNFs to utilize the OvS-DPDK FDP in addition to SR-IOV. In this post, we will demonstrate the steps required to deploy a CNF with additional OvS-DPDK's vhostuser ports. 

In summary, the steps we will walk through include:

1. Creating and deploying a MachineSet specifying additional fast datapath networks. The MachineSet adds fast datapath worker nodes to an existing OpenShift cluster.

2. Binding the vfio-pci driver to additional fast datapath networks using a MachineConfig specification.

3. Creating additional networks in the cluster using a NetworkAttachmentDefinition and bind those networks using host-devices.

Requisites

Before we get started with the deployment, we need to check for these system and network requirements:

• Red Hat OpenStack 16.2 deployed with bare-metal compute nodes having OvS-DPDK FastData path enabled.

• Red Hat OpenShift 4.9 (or 4.10) deployed on the deployed Red Hat OpenStack environment. The SR-IOV network operator is not required for using vhostuser ports.

• performance-addon-operator deployed with CPU isolation and huge pages, which should be applied to the worker node profile.

• Additional OvS-DPDK neutron networks created with the same configuration as other VNF applications running in the Red Hat OpenStack layer. For the following example, let's assume two VLAN-based OvS-DPDK networks are created with names "vhostuser1" and "vhostuser2". Follow the OSP NFV document for detailed steps.

Create Red Hat OpenShift Container Platform Worker Nodes with Vhostuser

You can create the OpenShift Container Platform worker nodes by following the MachineSet approach detailed in these docs. The MachineSet method allows customization of the worker node machine profile by specifying the additional network connections for a worker node in the MachineSet specification. 

An Ansible role has been created to facilitate the creation of the MachineSet configuration. Follow the README in the role to create inventory.yml, which contains all the required information to create a MachineSet with additional networks. 

all:
 hosts:
    localhost:
      ansible_connection: local
      ansible_python_interpreter: "{{ansible_playbook_python}}"

      number_of_replicas: 1

      cluster_metadata_path: "~/sos-fdp/build-artifacts/metadata.json"

      openstack_cloud: "openstack"

      nova_flavor: "ocp-worker"
      node_role: "worker"

      additional_networks:
      - name: "vhostuser1"
        name_suffix: "vhostuser1"
        tags:
          - uplink_input
        vnic_type: "normal"
        driver: "vfio-pci"
      - name: "vhostuser2"
        name_suffix: "vhostuser2"
        tags:
          - uplink_output
        vnic_type: "normal"
        driver: "vfio-pci"

Next, create the MachineSet via the role and deploy it. It will create the worker nodes with the additional vhostuser ports. The primary Kubernetes network port will be created using the default OpenStack network type.

$ git clone git@github.com:rh-nfv-int/shift-on-stack-machineset.git
$ cd shift-on-stack-machineset

# create inventory.yaml as show above

$ ansible-playbook play.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml

$ oc apply -f ./build/<infraID>-worker-machineset.yaml

Userspace Driver Binding on OpenShift Container Platform Worker nodes

CNFs that do packet processing on the datapath require userspace drivers to be bound to the interface inside the worker node VM. For DPDK-based CNF applications, the userspace driver vfio-pci should be bound to the interfaces so that the application can perform userspace packet processing.

OCP Worker nodes running on Red Hat CoreOS support vfio-pci driver to be used as the userspace driver. Additional configuration is required on the worker node to enable this support. Note that this is in Tech Preview and not currently supported in production. The vfio module on the created worker node should be loaded with additional flags to enable unsafe noiommu mode enable_unsafe_noiommu_mode=1. Here, MachineConfig can be created in order to create the modprobe conf file to load vfio with this option enabled.

kind: MachineConfig
apiVersion: machineconfiguration.openshift.io/v1
metadata:
 name: 99-vfio-noiommu
 labels:
   machineconfiguration.openshift.io/role: worker
spec:
 osImageURL: ''
 config:
   ignition:
     version: 2.2.0
   storage:
     files:
     - filesystem: root
       path: "/etc/modprobe.d/vfio-noiommu.conf"
       contents:
         source: data:text/plain;charset=utf-8;base64,b3B0aW9ucyB2ZmlvIGVuYWJsZV91bnNhZmVfbm9pb21tdV9tb2RlPTEK
         verification: {}
       mode: 0644

In order to bind the vhostuser interfaces to the vfio-pci driver, a set of scripts has been added to the Ansible role shift-on-stack-machineset, which creates MachineConfig resource with these scripts files. Once the MachineConfig is applied on the worker nodes, the vfio-pci driver will be bound to the interface on every reboot. The MachineConfig creation is part of the gen.yaml playbook, which will be executed in the next step.

Create Vhostuser OpenShift Container Platform Network

Additional networks can be attached to the CNF application using the NetworkAttachmentDefintion resource. The NetworkAttachmentDefinition utilizes the host-device CNI plugin to attach vhostuser-based additional interfaces to the CNF application POD. The next section provides a sample definition of the NetworkAttachmentDefintion resource created with a specific PCI address.

apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
 name: vhostuser1
 namespace: example-cnf
spec:
 config: '{ "cniVersion": "0.3.1", "name": "hostonly", "type": "host-device", "pciBusId": "0000:00:04.0", "ipam": { } }'

$ cd shift-on-stack-machineset

$ ansible-playbook gen.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml

$ oc apply -f build/<infraID>-<net>-netattach.yaml
$ oc apply -f build/<infraID>-machine-config.yaml

The NetworkAttachmentDefinition resources for vhostuser1 and vhostuser2 networks will be created on the targeted namespace example-cnf.

$ oc -n example-cnf get net-attach-def
NAME         AGE
vhostuser1   5h44m
vhostuser2   5h44m

Create CNF Application with Vhostuser 

Finally, let’s create a CNF application using testpmd by adding the created networks by using this template format.

apiVersion: v1
kind: Pod
metadata:
 name: testpmd
 namespace: example-cnf
 annotations:
   k8s.v1.cni.cncf.io/networks: "vhostuser1,vhostuser2"
spec:
 containers:
 - name: testpmd
   command: ["/bin/sh"]
   args: ["-c", "testpmd -l $(taskset -pc 1 | cut -d: -f2) --in-memory -w 00:04.0 -w 00:05.0 --socket-mem 1024 -n 4 -- --nb-cores=1 --auto-start --forward-mode=mac --stats-period 10"]
   image: registry.redhat.io/openshift4/dpdk-base-rhel8:v4.6
   securityContext:
     privileged: true
     runAsUser: 0
   resources:
     requests:
       memory: 1000Mi
       hugepages-1Gi: 3Gi
       cpu: '3'
     limits:
       hugepages-1Gi: 3Gi
       cpu: '3'
       memory: 1000Mi
   volumeMounts:
     - mountPath: /dev/hugepages
       name: hugepage
       readOnly: False
 volumes:
 - name: hugepage
   emptyDir:
     medium: HugePages

NOTE: For shift-on-stack CNF applications, the security context "privileged: true" is mandatory to allow the application to access PCI devices. This is a limitation of the current version.

Conclusion

As many of the network function vendors are transitioning from the VNF to CNF era, the shift-on-stack deployment model with vhostuser provides a head start in creating a NFV-ready environment to develop and integrate CNFs. 

This deployment strategy gives an opportunity for service providers to experience CNFs on the existing Red Hat OpenStack Platform.

Learn more about our range of solutions that can help telco providers with flexible deployment configurations in this blog post.