OVS-DPDK: Migrating to vhostuser socket mode in Red Hat OpenStack

In the Newton release, the default vhostuser mode in Open vSwitch (OvS) is dpdkvhostuser. In Red Hat OpenStack Platform 10 GA, and subsequent updates till 29th June, the default vhostuser mode is dpdkvhostuser. With the latest update to OSP10 (post 29th June 2018), the default mode has been changed to dpdkvhostuserclient. This post provides the information on vhostuser migration and verifying the vhostuser modes of the VMs created with dpdkvhostuser mode.

In order to understand the difference between the two modes and the advantage of moving to dpdkvhostuserclient mode, read the OvS documentation onvhostuser modes. In short, vhostuser allows Qemu to fetch/put network data to OvS-DPDK without overloading Qemu with the translation. And the vhostuser socket is a UNIX domain socket, created to establish the communication between Qemu and OvS-DPDK. This communication follows a specific messaging format detailed in theQemu's vhost user document.

In the dpdkvhostuserclient mode, Qemu acts as the server and creates the vhostuser socket. And OvS acts as client which connects to the created socket. Using this mode, crashes/restarts on the OvS-DPDK application does not require to restart the VM, the client will establish the connection again after starting again.

The steps for migration existing VMs with server to client mode in OvS is detailed in the minor update documentation, along with the limitations. Let’s see how and where to check the vhostuser mode information in both Qemu and OvS. The following sections shows the difference that these two modes create on a compute node in a TripleO deployment. This is applicable since the Red Hat OpenStack Platform 10 release.

dpdkvhostuser

In the dpdkvhostuser mode, the socket will be created by OvS in the /var/run/openvswitch directory. The OvS service file should be patched to make OvS run with qemu as group ownership. This allows the Qemu, which is the client, to connect to the socket.

This snippet shows that the vhostuser socket is created with the ownership as root:qemu in the default directory.

This snippet shows that the vhostuser socket's port details attached to the br-int bridge, which also shows the mode of the socket.

# Partial output
$ ovs-vstl show
   Bridge br-int
       Port "vhu1879c31a-fa"
           tag: 1
           Interface "vhu1879c31a-fa"
               type: dpdkvhostuser

The OvS process runs with the patched ownership. This snippet shows that the ovs-vswitchd process runs with the root:qemu ownership.

# Partial output
$ ps ax -o user,group,pid,command,args | grep -e ovsdb-server -e \ 
  ovs-vswitchd
root     root  11770 ovsdb-server /etc/openvswit
root     qemu  11806 ovs-vswitchd unix:/var/run/

This snippet shows the extract from the virsh xml of a VM with vhostuser socket is connected in the client mode.

# Partial output
$ virsh dumpxml instance-00000002
   <interface type='vhostuser'>
     <mac address='fa:16:3e:4b:ea:c4'/>
     <source type='unix' path='/var/run/openvswitch/vhu1879c31a-fa' \ 
mode='client'/>
     <model type='virtio'/>
     <alias name='net0'/>
     <address type='pci' domain='0x0000' bus='0x00' slot='0x03' \ 
function='0x0'/>
   </interface>

The libvirt logs shows the socket and its mode attached to the VM. If there is any failure in the migration, this log will help to understand the mode of the vhostuser socket attached to the VM.

# Log from /var/log/libvirt/qemu/instance-00000002.log file
-chardev socket,id=charnet0,path=/var/run/openvswitch/vhueacd1f04-7a

The process associated with the VM runs with qemu:qemu ownership.

# Partial output

$ ps ax -o user,group,pid,command,args | grep -e qemu-kvm
qemu     qemu 166572 /usr/libexec/qemu-kvm -name \ 
 /usr/libexec/qemu-kvm -name guest=instance-00000002,debug-threads=on

dpdkvhostuserclient

In the dpdkvhostuserclient mode, the socket will be created by Qemu in the /var/lib/vhost_sockets directory. A new group hugetlbfs has been created which will be used to share the socket between Qemu and OvS. The group value in the /etc/libvirt/qemu.conf will be configured with the new group hugetlbfs. All the VMs created then on will run with qemu:hugetlbfs ownership. OvS will run with the ownership openvswitch:hugetlbfs, which has been introduced from OvS 2.8 version onwards. The configuration OVS_USER_ID in the file /etc/sysconfig/openvswitch, will configure the ownership of the OvS process that will run on the node.

The ovsdb-server service file will change the ownership of the /var/run/openvswitch and ovs-vswitchd service file will change the ownership of /dev/hugepages to the value configured using OVS_USER_ID.

Below snippets demonstrate this behavior of change in the ownership of the files and process and other configuration compare to the older vhostuser mode.

$ ll /var/lib/vhost_sockets/vhu28b691d4-0e
srwxrwxr-x. 1 qemu hugetlbfs 0 Jun 21 12:00 \ 
 /var/lib/vhost_sockets/vhu28b691d4-0e

# Partial output
$ ovs-vstl show
  Port "vhu28b691d4-0e"
      tag: 1
      Interface "vhu28b691d4-0e"
          type: dpdkvhostuserclient
          options: {vhost-server-path="/var/lib/vhost_sockets/vhu28b691d4-0e"}

# Partial output
$ ps ax -o user,group,pid,command,args | grep -e ovsdb-server -e \ 
 ovs-vswitchd
openvsw+ hugetlb+    1691 ovsdb-server /etc/openvswit ovsdb-server
openvsw+ hugetlb+    1773 ovs-vswitchd unix:/var/run/ ovs-vswitchd

This snippet shows the extract from the virsh XML of a VM with Qemu as server.

# Partial output
$ virsh dumpxml instance-00000003
   <interface type='vhostuser'>
     <mac address='fa:16:3e:6d:49:e8'/>
     <source type='unix' \ 
  path='/var/lib/vhost_sockets/vhu28b691d4-0e' mode='server'/>
     <target dev='vhu28b691d4-0e'/>
     <model type='virtio'/>
     <alias name='net0'/>
     <address type='pci' domain='0x0000' bus='0x00' \ 
  slot='0x03' function='0x0'/>
   </interface>

# Log from /var/log/libvirt/qemu/instance-00000003.log file
-chardev socket,id=charnet0,path=/var/lib/vhost_sockets/vhu28b691d4-0e,server

# Partial output
$ ps ax -o user,group,pid,command,args | grep -e qemu-kvm
qemu     hugetlb+ 152317 /usr/libexec/qemu-kvm -name \ 
  /usr/libexec/qemu-kvm -name guest=instance-00000003,debug-threads=on

A note about group IDs (GIDs)

In Newton-based deployment, all the OpenStack services are deployed as systemd services, whereas on Queens based deployment, all the OpenStack services will be deployed using the containers.

But Open vSwitch still runs as the systemd service, which requires the vhostuser socket created by Qemu (running as nova_libvirt container) to share the same group id with the OvS running the host.

In order to facilitate the sharing of vhostuser between host and Kolla-based nova_libvirt container, the GID of hugetlbfs in the host has been aligned with the GID value of the hugetlbfs in the Kolla container. Also this allows us to enable the Newton-based deployment to upgrade to Queens without much difficulty. The GID value of the hugetlbfs group is configured as 42477.

Allowing time to migrate

Red Hat OpenStack supports both vhost user client and server mode, as some users may need some time to migrate to vhost user client mode. That said, Red Hat is recommending vhost user client mode as it has only advantages compared to server mode, and the corresponding code still exist to allow some time for users to migrate:

lib/netdev-dpdk.c: VLOG_WARN_ONCE("dpdkvhostuser ports are considered deprecated; "
lib/netdev-dpdk.c: "please migrate to dpdkvhostuserclient ports.");

This warning, and the dpdkvhostuser code, are very likely to be removed from the OVS 2.11 release upstream sometime after spring 2019.