[vfio-users] KVM internal error. Suberror: 1, emulation failure problem

Fri May 6 15:30:02 UTC 2016

thanks,Alex.

I use the same config to launch the VMs in the same time.Card with PCI address 02:00.0 and the 03:00.0 works fine, 82:00.0 and 83:00.0 are not.

I just check the BIOS settings.Configuration of PCI-e device is the same.

here is the lspci command output:

        02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
        Subsystem: LeadTek Research Inc. Device [107d:2762]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
        .......
        03:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
        Subsystem: LeadTek Research Inc. Device [107d:2762]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
        ......
        82:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
        Subsystem: LeadTek Research Inc. Device [107d:2762]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau
        ......
        83:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM204 [GeForce GTX 970] [10de:13c2] (rev a1)
        Subsystem: LeadTek Research Inc. Device [107d:2762]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

I also edit the /usr/bin/qemu-kvm like this:

    #!/bin/sh
    result="$@"
    result=${result/02:00.0/02:00.0,x-vga=on}
    result=${result/03:00.0/03:00.0,x-vga=on}
    result=${result/82:00.0/82:00.0,x-vga=on}
    result=${result/83:00.0/83:00.0,x-vga=on}

    exec /usr/bin/qemu-system-x86_64 -machine accel=kvm $result

in F22,the original /usr/bin/qemu-kvm is a script:

    #!/bin/sh

    exec /usr/bin/qemu-system-x86_64 -machine accel=kvm $@
From: Alex Williamson
Date: 2016-05-06 23:12
To: xuemin.wen
CC: vfio-users
Subject: Re: [vfio-users] KVM internal error. Suberror: 1, emulation failure problem
On Fri, 6 May 2016 22:56:14 +0800
"xuemin.wen" <Kayv at qq.com> wrote:

> hello everyone,
>     I am trying to passthrough 4*GTX970 video card to individual VMs,here is the problem.
> 
>     the GTX970 with PCI address 02:00.0 and 03:00.0 can passthrough to the VMs,and works good.
>     
>     the GTX970 with PCI address 82:00.0 and 83:00.0 also can passthrough to the VMs,but the VMs just work for a few seconds with CPU usage 100%,and return a error:"KVM internal error. Suberror: 1 emulation failure"
> 
>     if i don't passtrough GTX 970,there is no error. So I think the issue with the relevant VFIO,not the QEMU-KVM. 
>     
>     my host hardware is :
>         Supermicro 1028GQ-TR system include X10DGQ motherboard
>         XEON E5 2630v3 * 2
>         16GB RAM * 4
>         LEADTEK GTX970 * 4
>         INTEL SSD S3510
> 
>     my host system is :
>         Fedora 22 server at 4.4.5-200.fc22.x86_64
>         QEMU emulator version 2.3.1 (qemu-2.3.1-12.fc22)
>         libvirt version: 1.2.13.2
> 
>     the log of GTX with PCI address 82:00.0 or 83:00.0 
> 
> 2016-05-06 13:37:10.197+0000: starting up libvirt version: 1.2.13.2, package: 2.fc22 (Fedora Project, 2016-01-21-01:06:26, buildvm-06.phx2.fedoraproject.org), qemu version: 2.3.1 (qemu-2.3.1-12.fc22)
> LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/bin/qemu-kvm -name instance-000002ef -S -machine pc-i440fx-2.3,accel=kvm,usb=off -cpu Haswell-noTSX,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,kvm=off -m 4096 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1 -uuid 92af0144-107e-48e1-bed1-ad04fac1faea -smbios type=1,manufacturer=Fedora Project,product=OpenStack Nova,version=2014.2.4-1.fc22,serial=26348ff0-e2d1-459f-a026-ec33b800e05c,uuid=92af0144-107e-48e1-bed1-ad04fac1faea -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/instance-000002ef.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/nova/insta!
nces/92af0144-107e-48e1-bed1-ad04fac1faea/disk,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=27,id=hostnet0,vhost=on,vhostfd=29 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:30:0d:09,bus=pci.0,addr=0x2 -chardev file,id=charserial0,path=/var/lib/nova/instances/92af0144-107e-48e1-bed1-ad04fac1faea/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -device vfio-pci,host=83:00.0,id=hostdev0,bus=pci.0,addr=0x4 -device vfio-pci,host=83:00.1,id=hostdev1,bus=pci.0,addr=0x5 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on
> char device redirected to /dev/pts/3 (label charserial1)
> KVM internal error. Suberror: 1
> emulation failure
> EAX=00000000 EBX=00000000 ECX=00000000 EDX=00000000
> ESI=00000000 EDI=00000000 EBP=00000000 ESP=00006de8
> EIP=0006fdff EFL=00010006 [-----P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
> ES =0000 00000000 ffffffff 00809300
> CS =3020 00030200 ffffffff 00809b00
> SS =0000 00000000 ffffffff 00809300
> DS =0000 00000000 ffffffff 00809300
> FS =0000 00000000 ffffffff 00809300
> GS =0000 00000000 ffffffff 00809300
> LDT=0000 00000000 0000ffff 00008200
> TR =0000 00000000 0000ffff 00008b00
> GDT=     000f6b70 00000037
> IDT=     00000000 000003ff
> CR0=00000010 CR2=00000000 CR3=00000000 CR4=00000000
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
> DR6=00000000ffff0ff0 DR7=0000000000000400
> EFER=0000000000000000
> Code=00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> qemu: terminating on signal 15 from pid 1500
> 2016-05-06 13:44:57.035+0000: shutting down

Seems like you're not using OVMF and you're not enabling VGA, so your
config is in some weird state where the VGA option ROM gets executed
but VGA resources don't actually get routed to the device.  Not
terribly predictable what might happen in that case.  If you have other
VMs working with this config, you might try dumping the ROM from those
cards and passing it as a file to these cards.  Maybe the host BIOS
does strange things in enabling ROMs on the second root bridge, or
doesn't enable them at all.  With GeForce cards, you preferably want to
use an OVMF VM or you want to enable x-vga=on with the vfio-pci
device.  Thanks,

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20160506/ae9bfcb3/attachment.htm>