[vfio-users] vfio help needed (nvidia GTX 750 Ti persistent code 43)
Alex Williamson
alex.williamson at redhat.com
Wed Sep 16 15:53:28 UTC 2015
On Wed, 2015-09-16 at 23:10 +0900, Mogliii wrote:
> Hello,
>
> I have been trying to get vga passthrough working for quite some time
> now. I even ordered a new graphics card just for that purpose.
>
> I explain my setup and configuration below.
>
> My current problem: I can install Windows 8.1 with uefi (gpt partition
> layout) and using virt-manager. Then, I shutdown the VM, remove the
> spice and graphics, and add the pcie MSI GTX 750 Ti Gaming (it has a
> legacy and uefi bios; uefi bios selected by physical switch). When I
> then boot-up the VM, I see the OVMF bios on the physical screen attached
> to the GTX750 Ti (sometimes I get graphics glitches i.e. all white) and
> Windows 8.1 boots into 800x600. I see that a Nvidia GTX750Ti is
> recognized in the device manager. After installing driver (I tried
> 335.23, 340.52, 341.44 and most recent) the gpu is shown with yellow
> exclamation mark with error code 43 and I have to restart. After
> restart, I see again the bios and the boot splash, but then the screen
> goes into power save mode because it does not get any signal just when
> the login screen is supposed to appear.
> I can login via vnc and see that the gpu has error code 43.
> De-installing the driver and restarting I will go back to a 800x600
> desktop on my physical screen.
Hmm, that's already strange, generally if the card gets code 43 it
continues to run in that low resolution mode you had before installing
the driver, not loss of signal.
> Any help with my setup would be immensely appreciated. I have another
> spare ASUS HD6570, but it does not have uefi bios. I am not sure if I
> would need i915 patch in that case? Does that patch only apply to Intel
> CPUs?
As others have reported, you can run non-UEFI cards with OVMF, but if it
works, the screen won't light up until the OS is nearly done booting.
> Is the 00:02.2 pci device the problem?
>
> I would really appreciate any hint as I currently exhausted all options
> I see and I would like to avoid spending more money on video cards that
> I'm not sure I can even use.
>
> ==========================
> SETUP
> =========
>
> [[[ 1. Physical setup
> Asus A88x-PRO
> AMD A10-7850K (on-board GPU used for linux host with radeon driver
> (fglrx not installed))
> 12 GB memory
> MSI GTX 750 Ti Gaming (bios can be switched between legacy/uefi by
> hardware switch; using uefi)
>
> GPU is installed in middle pcie 16x slot (black, not golden) due to
> iommu grouping (see below)
>
>
> [[[ 2. OS
> UBuntu 15.04 64 bit, unpatched
> $ uname -a
> Linux host 3.19.0-28-generic #30-Ubuntu SMP Mon Aug 31 15:52:51 UTC 2015
> x86_64 x86_64 x86_64 GNU/Linux
>
>
> [[[ 3. Relevant packages (not sure if these are all)
> virt-manager 1:1.2.1-1~getdeb1 (get deb ppa)
> qemu 1:2.2+dfsg-5expubuntu9.4
> libvirt-bin 1.2.12-0ubuntu14.2
> qemu-kvm 1:2.2+dfsg-5expubuntu9.4
>
> [[[ 4. Bios settings
> Set to boot from on board GPU. AMD-V and IOMMU activated. Legacy boot
> for ubuntu host
>
> [[[ 5. iommu grouping (iommu.sh output)
> ### Group 2 ###
> 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1424
> 00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1425
> 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce
> GTX 750 Ti] (rev a2)
> 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
>
> [[[ 6. pci identifier
> 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM107
> [GeForce GTX 750 Ti] [10de:1380] (rev a2)
> Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102]
> Kernel driver in use: pci-stub
> 01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbc] (rev a1)
> Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3102]
> Kernel driver in use: pci-stub
>
> [[[ 7. /proc/cmdline
> BOOT_IMAGE=/vmlinuz-3.19.0-28-generic root=/dev/mapper/main-root ro
> rootflags=subvol=@ quiet splash nomdmonddf nomdmonisw amd_iommu=on
> iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1
> pci-stub.ids=10de:0fbc,10de:1380 vt.handoff=7
>
> [[[ 8. /etc/initramfs-tools/modules
> pci_stub ids=10de:0fbc,10de:1380
>
> [[[ 9. /etc/modules
> lp
> rtc
> pci_stub
> vfio
> vfio_iommu_type1
> vfio_pci
> kvm
> kvm_amd
>
> [[[ 10. # dmseg | grep pci-stub
> [ 1.263269] pci-stub: add 10DE:0FBC sub=FFFFFFFF:FFFFFFFF
> cls=00000000/00000000
> [ 1.263287] pci-stub 0000:01:00.1: claimed by stub
> [ 1.263296] pci-stub: add 10DE:1380 sub=FFFFFFFF:FFFFFFFF
> cls=00000000/00000000
> [ 1.263304] pci-stub 0000:01:00.0: claimed by stub
I'm always suspicious when pci-stub is a module that it's actually
preventing use of the device, this adds some comfort, but still a little
bit of suspicion.
> [[[ 11. /usr/windows.sh (executed before running vm. Afterwards lspci
> -nnk shows "vfio-pci" as kernel driver in use for 01:00.0 and 01:00.1)
> #!/bin/bash
>
> configfile=/etc/vfio_pci1.cfg
> vfiobind() {
> dev="$1"
> vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
> device=$(cat /sys/bus/pci/devices/$dev/device)
> if [ -e /sys/bus/pci/devices/$dev/driver ]; then
> echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
> fi
> echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
> }
> modprobe vfio-pci
> cat $configfile | while read line;do
> echo $line | grep ^# >/dev/null 2>&1 && continue
> vfiobind $line
> done
> exit 0
>
> [[[ 12. /etc/vfio_pci1.cfg
> 0000:01:00.0
> 0000:01:00.1
Gag, I hate these scripts, especially when you're using libvirt for the
VM, 100% useless.
> [[[ 13. OVMF-bios
> Got from sourceforge
> (edk2.git-ovmf-x64-0-20150910.b1206.g5abc2a7.noarch.rpm) and extracted.
> Then, made individual copy of OVMF-pure-efi.fd (md5
> fbb13f2d4be2f9d72472628f885f5cb4 ) for VM.
>
> [[[ 14. Created VM in virt-manager using the following configuration
> (configuration shown after installation, i.e. no cdroms, but before
> attaching pci)
> <domain type='kvm'>
> <name>win81</name>
> <uuid>redacted</uuid>
> <memory unit='KiB'>4194304</memory>
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static'>2</vcpu>
> <os>
> <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
> <loader type='rom'>/usr/share/ovmf/OVMF-pure-efi_win81.fd</loader>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> <kvm>
> <hidden state='on'/>
> </kvm>
> </features>
> <cpu mode='host-passthrough'>
> </cpu>
> <clock offset='localtime'>
> <timer name='rtc' tickpolicy='catchup'/>
> <timer name='pit' tickpolicy='delay'/>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/usr/bin/qemu-system-x86_64</emulator>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw' cache='none' io='native'/>
> <source dev='/dev/main/Win7xen'/>
> <target dev='vda' bus='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
> </disk>
> <controller type='usb' index='0' model='ich9-ehci1'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x7'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci1'>
> <master startport='0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0' multifunction='on'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci2'>
> <master startport='2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x1'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci3'>
> <master startport='4'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'/>
> <controller type='virtio-serial' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </controller>
> <interface type='bridge'>
> <mac address='52:54:00:ab:b6:10'/>
> <source bridge='br1'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
> </interface>
> <input type='mouse' bus='ps2'/>
> <input type='keyboard' bus='ps2'/>
> <graphics type='spice' autoport='yes'/>
> <video>
> <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x02'
> function='0x0'/>
> </video>
> <hostdev mode='subsystem' type='usb' managed='yes'>
> <source>
> <vendor id='0x0603'/>
> <product id='0x1605'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='yes'>
> <source>
> <vendor id='0x05ac'/>
> <product id='0x1006'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='yes'>
> <source>
> <vendor id='0x05ac'/>
> <product id='0x024f'/>
> </source>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
> </memballoon>
> </devices>
> </domain>
>
> [[[ 15. After installation and installation of updates, shut down the VM
>
> [[[ 16. Attached PCI devices and removed video and display spice. Below
> is virsh configuration after attaching pcie passthrough.
> <domain type='kvm'>
> <name>win81</name>
> <uuid>redacted</uuid>
> <memory unit='KiB'>4194304</memory>
> <currentMemory unit='KiB'>4194304</currentMemory>
> <vcpu placement='static'>2</vcpu>
> <os>
> <type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
> <loader type='rom'>/usr/share/ovmf/OVMF-pure-efi_win81.fd</loader>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> <kvm>
> <hidden state='on'/>
> </kvm>
> </features>
> <cpu mode='host-passthrough'>
> </cpu>
> </clock>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>restart</on_crash>
> <pm>
> <suspend-to-mem enabled='no'/>
> <suspend-to-disk enabled='no'/>
> </pm>
> <devices>
> <emulator>/usr/bin/qemu-system-x86_64</emulator>
> <disk type='block' device='disk'>
> <driver name='qemu' type='raw' cache='none' io='native'/>
> <source dev='/dev/main/Win7xen'/>
> <target dev='vda' bus='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x08'
> function='0x0'/>
> </disk>
> <controller type='usb' index='0' model='ich9-ehci1'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x7'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci1'>
> <master startport='0'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x0' multifunction='on'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci2'>
> <master startport='2'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x1'/>
> </controller>
> <controller type='usb' index='0' model='ich9-uhci3'>
> <master startport='4'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x06'
> function='0x2'/>
> </controller>
> <controller type='pci' index='0' model='pci-root'/>
> <controller type='virtio-serial' index='0'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x05'
> function='0x0'/>
> </controller>
> <interface type='bridge'>
> <mac address='52:54:00:ab:b6:10'/>
> <source bridge='br1'/>
> <model type='virtio'/>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x04'
> function='0x0'/>
> </interface>
> <hostdev mode='subsystem' type='usb' managed='yes'>
> <source>
> <vendor id='0x0603'/>
> <product id='0x1605'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='yes'>
> <source>
> <vendor id='0x05ac'/>
> <product id='0x1006'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='usb' managed='yes'>
> <source>
> <vendor id='0x05ac'/>
> <product id='0x024f'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
This managed='yes' option means that libvirt will do the bind to
vfio-pci for you.
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x03'
> function='0x0'/>
> </hostdev>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
> </source>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x09'
> function='0x0'/>
> </hostdev>
> <memballoon model='virtio'>
> <address type='pci' domain='0x0000' bus='0x00' slot='0x07'
> function='0x0'/>
> </memballoon>
> </devices>
> </domain>
>
> [[[ 17. What else did I try
> - Different nvidia drivers (see top of email)
> - Attaching 00:02.2 as well, but cannot do vfio-pci assignement so failed
Never assign bridges
> # virsh nodedev-list --tree
> +- pci_0000_00_02_0
> +- pci_0000_00_02_2
> | |
> | +- pci_0000_01_00_0
> | +- pci_0000_01_00_1
> |
> +- pci_0000_00_03_0
>
> # lspci -nnk (extract)
> 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device
> [1022:1424]
> 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device
> [1022:1425]
> Kernel driver in use: pcieport
>
> - Reinstall the whole VM with seabios and using the GTX 750 Ti with
> legacy bios (physical switch on card). But no picture was shown on
> screen after attaching, not even the bios.
> - Tried to use q35 emulation, but not successful. It does not work with
> ovmf? I.e. I cannot use the qemu:cmd or ioh3420.
q35 is a red herring for a Windows guest, you don't need it. Look in
the libvirt log for the domain and report the actually QEMU commandline
used to make sure there aren't any hyperv options sneaking past.
More information about the vfio-users
mailing list