Using Hooks in Red Hat Enterprise Virtualization

As a Solutions Architect, I enjoy creating and adding custom configurations to my Red Hat Enterprise Virtualization(RHEV) environment using a feature called hooks. A hook is a custom script that executes at a certain point during a RHEV event. You can attach scripts to several events. To see the full list of RHEV hooks, do a directory listing of “/usr/libexec/vdsm/hooks” on a RHEV hypervisor and you will see the below list.

To illustrate the process of using hooks, I am going to provide storage to my virtual cluster by adding CONNECTED_MODE to the hypervisors. This customized configuration will enhance performance of the Infiniband connected iSCSI storage server. Why do I need a hook for this process? RHEV supports iSCSI and Infiniband without the need for customized scripts. At the moment, Red Hat Enterprise Virtualization Manager (RHEV-M) only supports DATAGRAM_MODE for Infiniband, with a maximum MTU of 2044. If a user requires a higher number of MTU, we need to use the CONNECTED_MODE for the Infiniband iSCSI network for best performance.

Configuring RHEV Networking to use the Infiniband Device for iSCSI Storage

I will assume that you have an Infiniband switch, cables, and cards already installed in your hypervisors, and that you have a iSCSI storage server connected to the Infiniband network using IpoIB. You have also created the iSCSI logical network in RHEV-M’s Networks tab. To do the configuration, you need to follow the following steps:

1. In Maintenance mode, install the following packages on the hypervisor
   1. [root@rhev2 ~]# yum -y install rdma ibutils libibverbs-utils infiniband-diags
   2. [root@rhev2 ~]# echo “alias ib0 ib_ipoib” > /etc/modprobe.d/ipoib.conf
2. Reboot.
3. Check the status of your Infiniband device with the following two commands: “ibstat” and “ibstatus”. The device should be physically connected and you will see the following results:

[root@rhev2 ~]# ibstat
CA 'mthca0'
CA type: MT25204
Number of ports: 1
Firmware version: 1.1.0
Hardware version: a0
Node GUID: 0x0005ad00000c5a30
System image GUID: 0x0005ad00000c5a33
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 7
LMC: 0
SM lid: 1
Capability mask: 0x02590a68
Port GUID: 0x0005ad00000c5a31
Link layer: InfiniBand
===========================
[root@rhev2 ~]# ibstat
CA 'mthca0'
CA type: MT25204
Number of ports: 1
Firmware version: 1.1.0
Hardware version: a0
Node GUID: 0x0005ad00000c5a30
System image GUID: 0x0005ad00000c5a33
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 7
LMC: 0
SM lid: 1
Capability mask: 0x02590a68
Port GUID: 0x0005ad00000c5a31
Link layer: InfiniBand

1. In RHEV-M, click on Hosts, then click on Network Interfaces screen. Here you can finish up the configuration.
2. Follow the steps in the Setup Host Networks' screen, assigning your ib0 device an IP address and your per-defined logical iSCSI network.
3. Look at your /etc/sysconfig/network-scripts/ifcfg-ib0 file on your RHEV hypervisor and see that VDSM has created the correct Infiniband device. You can confirm what MODE your Infiniband card is running in by looking at the contents of this file: /sys/class/net/ib0/mode

Through the following command: [root@rhev2 ~]# cat /sys/class/net/ib0/mode. The content should say datagram.

This is a working configuration and things are all running as expected, but we want to optimize our configuration. Follow the following steps to fine tune your configuration:

1. In RHEV-M, update the default MTU on our logical iSCSI network
   1. Click on the Networks tab -> Edit for your logical iSCSI network. Mine is called ibISCSI (as seen in the image below).
   2. Select the Custom MTU and enter 65520 or your desired MTU size as seen in Image I below.
      

      Image I

Perfect, but there is a problem. Since we told RHEV-M to change the MTU for that network, it attempted to do so by updating the /etc/sysconfig/network-scripts/ifcfg-ib0 file with an “MTU=65520”. However, we are still in DATAGRAM_MODE and our actual MTU is still set to 2044 since DATAGRAM_MODE only supports an MTU of 2044.

You can confirm this by running the following command on your hypervisor:
[root@rhev2 ~]# ip addr | grep ib0
7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
inet 192.168.103.173/24 brd 192.168.103.255 scope global ib0

RHEV-M will also show an error on our host as seen in Image II below. It will complain because the hypervisor ib0 device has an MTU of 2044 and the logical iSCSI network in the DATACENTER has been configured to have an MTU of 65520.

Image II

Even if we try to use the RHEV-M to fix the error, there are currently no Infiniband mode options in the Network Interfaces → Setup Host Networks drag and drop wizard tab as seen in Image III below.

Image III

The fix is straightforward and a great use case for RHEV hooks. We need to convince VDSM to add two simple options to our /etc/sysconfig/network-scripts/ifcfg-ib0 file.

"TYPE=Infiniband"

"CONNECTED_MODE=yes"

If you look at the network-related events in /usr/libexec/vdsm/hooks you will currently find three.

after_network_setup

after_network_setup_fail

before_network_setup

We need to decide where to install our hook script. Since our hook script will be modifying the /etc/sysconfig/network-scripts/ifcfg-ib0 file, we cannot use the before_network_setup event, because it will be overwritten by VDSM when the network starts. The after_network_setup_fail would be useful for failure recovery or notification type scripts. The only event left is the after_network_setup event.

Below is a simple bash script written to update the /etc/sysconfig/network-scripts/ifcfg-ib0 with the appropriate settings to allow the device to support CONNECTED_MODE. It has some basic error checking and as you can see, is hard coded to the ib0 device only. Hook scripts need to return either a 1 for an error or 0 for success. Once we update the file we bounce the device for the new options to take effect. This is done only once, when the hook script runs.

#!/bin/bash
# SCRIPT Name: 01-ib-connected-mode.sh
# DESCRIPTION: VDSM hook to force Infiniband mode from default datagram to connected mode
# REQUIREMENTS: Configure MTU in Network tab of RHEV-M with Custom MTU - should # be 66520 for CONNECTED_MODE
# LOCATION: /usr/libexec/vdsm/hooks/after_network_setup
if ! grep -Fxq connected /sys/class/net/ib0/mode
then
echo "TYPE=Infiniband" >> /etc/sysconfig/network-scripts/ifcfg-ib0
echo "CONNECTED_MODE=yes" >> /etc/sysconfig/network-scripts/ifcfg-ib0
/usr/sbin/ifdown ib0
/usr/sbin/ifup ib0
if grep -Fxq connected /sys/class/net/ib0/mode
then
return 0
else
return 1
fi
else
return 0
fi

Then follow the following steps:

1. Copy the hook into the following folder on your hypervisor while in maintenance mode: /usr/libexec/vdsm/hooks/after_network_setup
   MAKE SURE THE SCRIPT IS EXECUTABLE!
   [root@rhev2 ~]# chmod +x /usr/libexec/vdsm/hooks/after_network_setup/01-ib-connected-mode.sh
2. Reboot the server to make sure everything starts up as expected.
3. Once your hypervisor host has been activated, you should get the following output from these two command line commands on your RHEV hypervisor:

[root@rhev2 ~]# cat /sys/class/net/ib0/mode
connected
and
[root@rhev2 ~]# ip addr | grep ib0
7: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP qlen 256
inet 192.168.103.173/24 brd 192.168.103.255 scope global ib0

If you look at RHEV-M network settings, the iSCSI device has successfully synced as shown below in image IV.

Image IV

You can see  in Image V below what Host Hooks have been applied to your hypervisor by selecting the host hypervisor and clicking on the Host Hooks tab for a list

.

Image V

The three 50_vmfex hooks are for supporting Cisco VM-FEX hardware. They were introduced into RHEV 3.6 and only execute when VM-FEX hardware is present and can be ignored or removed if using other hardware.

A lot is possible with hooks which allow you to easily extend features and functionality within RHEV. Note: hooks are not officially supported and users can get help in the thriving community forums. Stay tuned for my upcoming blogs posts, “Building a RHEL6 or RHEL7 iSCSI Server” and “Optimizing a RHEL7 iSCSI Server With DM-Cache”. Please follow the below links for more information:

• Download RHEV
• RHEV Documentation
• RHEL Infiniband modes can be found in section 9.6.2 of this document