[vfio-users] VFIO device assignment fails for LSI SAS2008 on Xeon E3-1200 v3

Mon May 9 22:46:46 UTC 2016

Hi all,

I just tried Linux 4.6.0-rc7 with qemu 2.6.0-rc4. Same error and behaviour. What the heck is this?

Best,
Damon

> On 09 May 2016, at 17:49, Damon Namod <msg at damon.at> wrote:
> 
> Hi Alex,
> 
> I tried a simple VM with the command you provided, the error remains the same:
> 
>    # sudo qemu-system-x86_64 -enable-kvm -m 2G -net none -monitor stdio -serial none -parallel none -vnc :1 -device vfio-pci,host=1:00.0,rombar=0 -S -cdrom Fedora-Live-Cinnamon-x86_64-23-10.iso
>    QEMU 2.0.0 monitor - type 'help' for more information
>    (qemu) c
>    (qemu) qemu-system-x86_64: vfio_dma_map(0x55cf105918b0, 0xfebd0000, 0x2000, 0x7f98a9d05000) = -14 (Bad address)
>    qemu: hardware error: vfio: DMA mapping failed, unable to continue
>    CPU #0:
>    EAX=00000103 EBX=80001804 ECX=00000103 EDX=00000cfc
>    ESI=00001800 EDI=00000000 EBP=00000018 ESP=00006d3c
>    EIP=7ffdf25d EFL=00000046 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>    ES =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>    CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>    SS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>    DS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>    FS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>    GS =0010 00000000 ffffffff 00c09300 DPL=0 DS   [-WA]
>    LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
>    TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
>    GDT=     000f6658 00000037
>    IDT=     000f6696 00000000
>    CR0=00000011 CR2=00000000 CR3=00000000 CR4=00000000
>    DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
>    DR6=00000000ffff0ff0 DR7=0000000000000400
>    EFER=0000000000000000
>    FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
>    FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
>    FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
>    FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
>    FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
>    XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
>    XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
>    XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
>    XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
> 
> This is what the kernel reports:
> 
>    May  9 22:15:16 dahost kernel: [112620.877983] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)
> 
>> SeaBIOS isn't too fond of it with the ROM enabled (hangs for a while after SeaBIOS version string[1]), but if I run it with rombar=0, it seems pretty happy, certainly no mapping faults.
> 
> The card is flashed to IT mode and the boot support has been disabled. The controller operated well with those options using legacy KVM device assignment. Passing ANY other (onboard) device using VFIO works just fine.
> 
>> What's the host kernel version?  Can you try something newer?  Thanks,
> 
> The host is a Ubuntu 14.04.4 running Linux 4.4. I could give 4.5 a shot if you think it's worth it.
> 
> Thanks,
> Damon
> 
>> On 09 May 2016, at 16:35, Alex Williamson <alex.l.williamson at gmail.com> wrote:
>> 
>> On Sun, May 8, 2016 at 7:24 PM, Damon Namod <msg at damon.at> wrote:
>> Hi Okky,
>> 
>> thanks for you response!
>> 
>> Passing the card to a Linux (Ubuntu 15.04) guest fails with the same message:
>> 
>>        qemu-system-x86_64: vfio_dma_map(0x5613dc154610, 0xfeb90000, 0x2000, 0x7f7976a09000) = -14 (Bad address)
>>        qemu: hardware error: vfio: DMA mapping failed, unable to continue
>> 
>> The M1015 is already flashed to IT mode and I have disable bios boot support to circumvent the issue you described. I already switched the PCIe slot but the symptoms are the same. Actually it's an interesting thought that it might have something to do with the controller itself. Any more thoughts on this? Could be the Xeon E3-1200 and the laking ACS support be an issue (I don't think so though)? Could be a different machine type than `pc` help?
>> 
>> I also tried the latest GIT version of `qemu-system-x86_64` but the error message remains the same. Is there any way to get some more details about the error? `dmesg` isn't much of a help as it doesn't contain any messages regarding my problem. The only error message I got is found inside `/var/log/libvirt/qemu/`...
>> 
>> I don't know what you've got going on here.  The closest thing I've got is this:
>> 
>> 02:00.0 Serial Attached SCSI controller [0107]: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0086] (rev 05)
>> 
>> SeaBIOS isn't too fond of it with the ROM enabled (hangs for a while after SeaBIOS version string[1]), but if I run it with rombar=0, it seems pretty happy, certainly no mapping faults.
>> 
>> Can you try just a very simple VM, something like:
>> 
>> sudo qemu-system-x86_64 -enable-kvm -m 2G -net none -monitor stdio -serial none -parallel none -vnc :1 -device vfio-pci,host=1:00.0,rombar=0 -S -cdrom Fedora-Live-Cinnamon-x86_64-23-10.iso
>> 
>> (or whatever ISO image you like that you can boot up and poke around in the VM a bit)
>> 
>> This will start in paused mode to a (qemu) prompt on the terminal.  Use 'c' to continue execution.  Connect VNC to display :1, minding firewalls if you're connecting from another system.
>> 
>> What's the host kernel version?  Can you try something newer?  Thanks,
>> 
>> Alex
>> 
>> [1] Oh, it actually did boot eventually and executed the HBA option ROM, just took a minute or two.
> 