Cloud-native architectures and containers can increase efficiency, performance, and agility for telecommunications service providers as they continue to find ways to compete in the market. Developers, architects, and others involved in the software development process can consider a shift-on-stack deployment model with vhostuser to integrate cloud-native network functions (CNFs).
In network functions virtualization (NFV) deployments, OpenShift can be deployed on top of OpenStack, but the deployment only supports exposing SR-IOV Virtual Functions to the CNF running on the OpenShift worker node (VM running on OpenStack).
As many NFV deployments support userspace fast datapath via OvS-DPDK, it would be advantageous for CNFs to utilize the OvS-DPDK FDP in addition to SR-IOV. In this post, we will demonstrate the steps required to deploy a CNF with additional OvS-DPDK's vhostuser ports.
In summary, the steps we will walk through include:
-
Creating and deploying a MachineSet specifying additional fast datapath networks. The MachineSet adds fast datapath worker nodes to an existing OpenShift cluster.
-
Binding the vfio-pci driver to additional fast datapath networks using a MachineConfig specification.
-
Creating additional networks in the cluster using a NetworkAttachmentDefinition and bind those networks using host-devices.
Requisites
Before we get started with the deployment, we need to check for these system and network requirements:
-
Red Hat OpenStack 16.2 deployed with bare-metal compute nodes having OvS-DPDK FastData path enabled.
-
Red Hat OpenShift 4.9 (or 4.10) deployed on the deployed Red Hat OpenStack environment. The SR-IOV network operator is not required for using vhostuser ports.
-
performance-addon-operator deployed with CPU isolation and huge pages, which should be applied to the worker node profile.
-
Additional OvS-DPDK neutron networks created with the same configuration as other VNF applications running in the Red Hat OpenStack layer. For the following example, let's assume two VLAN-based OvS-DPDK networks are created with names "vhostuser1" and "vhostuser2". Follow the OSP NFV document for detailed steps.
Create Red Hat OpenShift Container Platform Worker Nodes with Vhostuser
You can create the OpenShift Container Platform worker nodes by following the MachineSet approach detailed in these docs. The MachineSet method allows customization of the worker node machine profile by specifying the additional network connections for a worker node in the MachineSet specification.
An Ansible role has been created to facilitate the creation of the MachineSet configuration. Follow the README in the role to create inventory.yml, which contains all the required information to create a MachineSet with additional networks.
all: hosts: localhost: ansible_connection: local ansible_python_interpreter: "{{ansible_playbook_python}}" number_of_replicas: 1 cluster_metadata_path: "~/sos-fdp/build-artifacts/metadata.json" openstack_cloud: "openstack" nova_flavor: "ocp-worker" node_role: "worker" additional_networks: - name: "vhostuser1" name_suffix: "vhostuser1" tags: - uplink_input vnic_type: "normal" driver: "vfio-pci" - name: "vhostuser2" name_suffix: "vhostuser2" tags: - uplink_output vnic_type: "normal" driver: "vfio-pci"
Next, create the MachineSet via the role and deploy it. It will create the worker nodes with the additional vhostuser ports. The primary Kubernetes network port will be created using the default OpenStack network type.
$ git clone git@github.com:rh-nfv-int/shift-on-stack-machineset.git
$ cd shift-on-stack-machineset
# create inventory.yaml as show above
$ ansible-playbook play.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml
$ oc apply -f ./build/<infraID>-worker-machineset.yaml
Once the MachineSet is applied, worker nodes will be created. Wait for the worker node to be ready before proceeding to the next step.Userspace Driver Binding on OpenShift Container Platform Worker nodes
CNFs that do packet processing on the datapath require userspace drivers to be bound to the interface inside the worker node VM. For DPDK-based CNF applications, the userspace driver vfio-pci should be bound to the interfaces so that the application can perform userspace packet processing.
OCP Worker nodes running on Red Hat CoreOS support vfio-pci driver to be used as the userspace driver. Additional configuration is required on the worker node to enable this support. Note that this is in Tech Preview and not currently supported in production. The vfio module on the created worker node should be loaded with additional flags to enable unsafe noiommu mode enable_unsafe_noiommu_mode=1. Here, MachineConfig can be created in order to create the modprobe conf file to load vfio with this option enabled.
kind: MachineConfig apiVersion: machineconfiguration.openshift.io/v1 metadata: name: 99-vfio-noiommu labels: machineconfiguration.openshift.io/role: worker spec: osImageURL: '' config: ignition: version: 2.2.0 storage: files: - filesystem: root path: "/etc/modprobe.d/vfio-noiommu.conf" contents: source: data:text/plain;charset=utf-8;base64,b3B0aW9ucyB2ZmlvIGVuYWJsZV91bnNhZmVfbm9pb21tdV9tb2RlPTEK verification: {} mode: 0644
In order to bind the vhostuser interfaces to the vfio-pci driver, a set of scripts has been added to the Ansible role shift-on-stack-machineset, which creates MachineConfig resource with these scripts files. Once the MachineConfig is applied on the worker nodes, the vfio-pci driver will be bound to the interface on every reboot. The MachineConfig creation is part of the gen.yaml playbook, which will be executed in the next step.
Create Vhostuser OpenShift Container Platform Network
Additional networks can be attached to the CNF application using the NetworkAttachmentDefintion resource. The NetworkAttachmentDefinition utilizes the host-device CNI plugin to attach vhostuser-based additional interfaces to the CNF application POD. The next section provides a sample definition of the NetworkAttachmentDefintion resource created with a specific PCI address.
apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: vhostuser1 namespace: example-cnf spec: config: '{ "cniVersion": "0.3.1", "name": "hostonly", "type": "host-device", "pciBusId": "0000:00:04.0", "ipam": { } }'
$ cd shift-on-stack-machineset $ ansible-playbook gen.yaml -e cluster_metadata_path=~/sos-fdp/build-artifacts/metadata.json -i ./inventory.yaml $ oc apply -f build/<infraID>-<net>-netattach.yaml $ oc apply -f build/<infraID>-machine-config.yaml
The NetworkAttachmentDefinition resources for vhostuser1 and vhostuser2 networks will be created on the targeted namespace example-cnf.
$ oc -n example-cnf get net-attach-def NAME AGE vhostuser1 5h44m vhostuser2 5h44m
Create CNF Application with Vhostuser
Finally, let’s create a CNF application using testpmd by adding the created networks by using this template format.
apiVersion: v1 kind: Pod metadata: name: testpmd namespace: example-cnf annotations: k8s.v1.cni.cncf.io/networks: "vhostuser1,vhostuser2" spec: containers: - name: testpmd command: ["/bin/sh"] args: ["-c", "testpmd -l $(taskset -pc 1 | cut -d: -f2) --in-memory -w 00:04.0 -w 00:05.0 --socket-mem 1024 -n 4 -- --nb-cores=1 --auto-start --forward-mode=mac --stats-period 10"] image: registry.redhat.io/openshift4/dpdk-base-rhel8:v4.6 securityContext: privileged: true runAsUser: 0 resources: requests: memory: 1000Mi hugepages-1Gi: 3Gi cpu: '3' limits: hugepages-1Gi: 3Gi cpu: '3' memory: 1000Mi volumeMounts: - mountPath: /dev/hugepages name: hugepage readOnly: False volumes: - name: hugepage emptyDir: medium: HugePages
NOTE: For shift-on-stack CNF applications, the security context "privileged: true" is mandatory to allow the application to access PCI devices. This is a limitation of the current version.
Conclusion
As many of the network function vendors are transitioning from the VNF to CNF era, the shift-on-stack deployment model with vhostuser provides a head start in creating a NFV-ready environment to develop and integrate CNFs.
This deployment strategy gives an opportunity for service providers to experience CNFs on the existing Red Hat OpenStack Platform.
Learn more about our range of solutions that can help telco providers with flexible deployment configurations in this blog post.
Sobre os autores
Saravanan KR is a Principal Software Engineer at Red Hat.
Emilien is a French citizen living in Canada (QC) who has been contributing to OpenStack since 2011 when it was still a young project. While his major focus has been the installer, his impact has helped customers have a better experience when deploying, upgrading and operating OpenStack at large scale. Technical and team leader at Red Hat, he's developing leadership skills with passion for teamwork and technical challenges. He loves sharing his knowledge and often give talks to conferences.
Aaron Smith is a Senior Principal Software Engineer at Red Hat.
Navegue por canal
Automação
Últimas novidades em automação de TI para empresas de tecnologia, equipes e ambientes
Inteligência artificial
Descubra as atualizações nas plataformas que proporcionam aos clientes executar suas cargas de trabalho de IA em qualquer ambiente
Nuvem híbrida aberta
Veja como construímos um futuro mais flexível com a nuvem híbrida
Segurança
Veja as últimas novidades sobre como reduzimos riscos em ambientes e tecnologias
Edge computing
Saiba quais são as atualizações nas plataformas que simplificam as operações na borda
Infraestrutura
Saiba o que há de mais recente na plataforma Linux empresarial líder mundial
Aplicações
Conheça nossas soluções desenvolvidas para ajudar você a superar os desafios mais complexos de aplicações
Programas originais
Veja as histórias divertidas de criadores e líderes em tecnologia empresarial
Produtos
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Red Hat Cloud Services
- Veja todos os produtos
Ferramentas
- Treinamento e certificação
- Minha conta
- Suporte ao cliente
- Recursos para desenvolvedores
- Encontre um parceiro
- Red Hat Ecosystem Catalog
- Calculadora de valor Red Hat
- Documentação
Experimente, compre, venda
Comunicação
- Contate o setor de vendas
- Fale com o Atendimento ao Cliente
- Contate o setor de treinamento
- Redes sociais
Sobre a Red Hat
A Red Hat é a líder mundial em soluções empresariais open source como Linux, nuvem, containers e Kubernetes. Fornecemos soluções robustas que facilitam o trabalho em diversas plataformas e ambientes, do datacenter principal até a borda da rede.
Selecione um idioma
Red Hat legal and privacy links
- Sobre a Red Hat
- Oportunidades de emprego
- Eventos
- Escritórios
- Fale com a Red Hat
- Blog da Red Hat
- Diversidade, equidade e inclusão
- Cool Stuff Store
- Red Hat Summit