[vfio-users] VFIO device assignment fails for LSI SAS2008 on Xeon E3-1200 v3

[Damon Namod]

Hi all,

I'm trying to pass a LSI SAS2008 controller to a guest using VFIO. First of all, some details on my system:

    # lsb_release -d
    Description:    Ubuntu 14.04.4 LTS
    
    # uname -rs
    Linux 4.4.0-22-generic
    
    # cat /proc/cpuinfo | grep -m 1 "model name"
    model name      : Intel(R) Xeon(R) CPU E3-1225 v3 @ 3.20GHz
    
    # qemu-system-x86_64 --version
    QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.22), Copyright (c) 2003-2008 Fabrice Bellard
    
    # libvirtd --version
    libvirtd (libvirt) 1.2.2

I'm using libvirt/virsh to pass the device to a (freenas) guest and at first it looks like the domain would start just fine because virsh reports "Domain freeness started". The domain, however, crashes right after start with the following message from `/var/log/libvirt/qemu/freenas.log`. The full log is attached:

	qemu-system-x86_64: vfio_dma_map(0x55b0a94b63c0, 0xfeb90000, 0x2000, 0x7f22ca700000) = -14 (Bad address)
	qemu: hardware error: vfio: DMA mapping failed, unable to continue
	
The LSI controller is in IOMMU group 1 and has the id `01:00.0`. It shares the group with a PCI bridge:

    ------------------
    | IOMMU group  1 |
    ------------------
    00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
            Kernel driver in use: pcieport
    01:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [1000:0072] (rev 03)
            Kernel driver in use: vfio-pci
            
The full IOMMU group distribution is attached, too.

I'm currently assigning the device manually to vfio-pci. The relevant part of the domain's XML configuration is:

    <hostdev mode='subsystem' type='pci' managed='no'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>

The problem only seems to occur with the LSI controller `01:00.0`. Passing another device in a non-singleton IOMMU group, e. g. the Firewire device `05:03.0` in group 17, works fine (even with `managed='yes'`):

    ------------------
    | IOMMU group 17 |
    ------------------
    04:00.0 PCI bridge [0604]: Tundra Semiconductor Corp. Device [10e3:8113] (rev 01)
    05:03.0 FireWire (IEEE 1394) [0c00]: Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx] [104c:8023]
            Kernel driver in use: firewire_ohci
            
Previously, I used to pass the controller using KVM pci assignment but due to it's deprecation in Linux 4.2 I wanted to switch to VFIO. Generally, it seems to work on my system but not for the device in question. 

I'd appreciate any help.

Best,
Damon

-------------- next part --------------
A non-text attachment was scrubbed...
Name: libvirt-qemu-freenas.log
Type: application/octet-stream
Size: 7185 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160507/5e06603f/attachment.obj>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: iommu-groups.txt
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160507/5e06603f/attachment.txt>