[vfio-users] amd rx480 invalid rom contents

Sat Jan 21 23:15:41 UTC 2017

Hi all,

Hope you can help me with this. I've built a new pc specifically with 
vfio in mind but unfortunately I can't get it too work. My pc consists 
of an Asus X99-M motherboard, Intel E5-1620v4, AMD RX480 in PCIE slot 1 
(for Windows guest) and AMD RX460 in slot 3 (for Linux host). Both GPU's 
are connected to the same monitor, I will switch inputs by using 
ddccontrol.

What is probably important to know, the one thing I missed when 
searching components is that Asus doesn't offer a 'boot gpu in slot X 
first' option. So that means when my pc boots it uses the RX480 as 
display. I have tried both setting the ids using pci-stub as vfio-pci, 
and although I can get the vfio-pci driver correctly assigned to the 
RX480 it seems in both cases there is a problem with the ROM.

When I start my Windows guest, I can see a quite some CPU activity in 
virt-manager but the screen never turns on. The dmesg log says the 
following:
-------------------
[  216.982418] vgaarb: device changed decodes: 
PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  217.091942] device vnet0 entered promiscuous mode
[  217.092107] br0: port 2(vnet0) entered forwarding state
[  217.092123] br0: port 2(vnet0) entered forwarding state
[  218.067955] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19 at 0x270
[  218.067961] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b at 0x2d0
[  218.067965] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e at 0x370
[  218.068429] vfio-pci 0000:01:00.0: BAR 0: can't reserve [mem 
0xc0000000-0xcfffffff 64bit pref]
[  220.465312] kvm: zapping shadow pages for mmio generation wraparound
[  220.465835] kvm: zapping shadow pages for mmio generation wraparound
[  220.512593] vfio-pci 0000:01:00.0: Invalid ROM contents
[  220.512617] vfio-pci 0000:01:00.0: Invalid ROM contents
-------------------

So I tried Alex's rom-parser, but unfortunately that doesnt seem to work 
either for the RX480:
-------------------
# ./rom-parser /tmp/rx480.rom
Valid ROM signature found @0h, PCIR offset 24ch
	PCIR: type 0 (x86 PC-AT), vendor: 1002, device: 67df, class: 030000
	PCIR: revision 0, vendor revision: f32
Error, ran off the end
-------------------

For the RX460 it does work:
-------------------
# ./rom-parser /tmp/rx460.rom
Valid ROM signature found @0h, PCIR offset 26ch
	PCIR: type 0 (x86 PC-AT), vendor: 1002, device: 67ef, class: 030000
	PCIR: revision 0, vendor revision: f32
Valid ROM signature found @ec00h, PCIR offset 1ch
	PCIR: type 3 (EFI), vendor: 1002, device: 67ef, class: 030000
	PCIR: revision 0, vendor revision: 0
		EFI: Signature Valid, Subsystem: Boot, Machine: X64
	Last image
-------------------

I am running Linux Mint 18, which is basically Ubuntu 16.04.
Kernel: Linux 4.4.0-59-generic
Qemu: 1:2.5+dfsg-5ubuntu10.6
Ovmf: 0~20160408.ffea0a2c-2
Amdgpu: amdgpu-pro-16.50-362463

Maybe related, in dmesg I also get these errors (repeatedly):
-------------------
[   17.918696] pcieport 0000:00:02.0: AER: Multiple Corrected error 
received: id=0010
[   17.918705] pcieport 0000:00:02.0: PCIe Bus Error: 
severity=Corrected, type=Data Link Layer, id=0010(Receiver ID)
[   17.918707] pcieport 0000:00:02.0:   device [8086:6f04] error 
status/mask=00000040/00002000
[   17.918708] pcieport 0000:00:02.0:    [ 6] Bad TLP
-------------------

See below the output of 'lspci -tv'
-------------------
-+-[0000:ff]-+-0b.0  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 
v4/Xeon D R3 QPI Link 0/1
              +(shorted_but_a_lot_of_Intel_stuff)
  \-[0000:00]-+-00.0  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 
v4/Xeon D DMI2
              +-01.0-[03]--
              +-02.0-[02]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] 
Device 67ef
              |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] 
Device aae0
              +-03.0-[01]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] 
Device 67df
              |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] 
Device aaf0
              +-05.0  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 
v4/Xeon D Map/VTd_Misc/System Management
              +-05.1  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 
v4/Xeon D IIO Hot Plug
              +-05.2  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 
v4/Xeon D IIO RAS/Control Status/Global Errors
              +-05.4  Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 
v4/Xeon D I/O APIC
              +-11.0  Intel Corporation C610/X99 series chipset SPSR
              +-14.0  Intel Corporation C610/X99 series chipset USB xHCI 
Host Controller
              +-16.0  Intel Corporation C610/X99 series chipset MEI 
Controller #1
              +-19.0  Intel Corporation Ethernet Connection (2) I218-LM
              +-1a.0  Intel Corporation C610/X99 series chipset USB 
Enhanced Host Controller #2
              +-1b.0  Intel Corporation C610/X99 series chipset HD Audio 
Controller
              +-1c.0-[04]--
              +-1c.3-[05]----00.0  Intel Corporation I210 Gigabit Network 
Connection
              +-1c.4-[06]----00.0  ASMedia Technology Inc. ASM1142 USB 
3.1 Host Controller
              +-1d.0  Intel Corporation C610/X99 series chipset USB 
Enhanced Host Controller #1
              +-1f.0  Intel Corporation C610/X99 series chipset LPC 
Controller
              +-1f.2  Intel Corporation C610/X99 series chipset 6-Port 
SATA Controller [AHCI mode]
              \-1f.3  Intel Corporation C610/X99 series chipset SMBus 
Controller
-------------------

Are these problems caused by the fact that I can't set the RX460 as 
boot_vga? Or is the RX480 at fault here? Or is there maybe something 
else at play?

Would appreciate your help!

With kind regards, Pim