[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[vfio-users] USB controller pass-through only works once per host boot and blocks HDMI audio.



As of updating to kernel 4.5.4-1-vfio ( from 4.4.5-1-vfio ) am having a strange issue in qemu (2.6.0) with my USB controller pass-through only working once per host boot. Also interesting to note that my Nvidia HDMI audio only works when the USB controller is not. The Nvidia HDMI audio never worked at all until the USB controller pass-through stopped working.

Another change that happened at the same time as the kernel is I did a fresh install of my Windows 10 guest to current stable. Was some random insider build before that.

On VM shutdown I always get the following one or more times:

qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.

Steps to Reproduce:
  1. boot host system
  2. start qemu
    • USB host controller works here, HDMI audio does not
  3. shut down and start VM again
    • HDMI audio pass-through works now, USB host controller does not
All boots after this reflect state of step 6 until host system is rebooted

Pass-through PCI devices:

00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31) (prog-if 30 [XHCI])
Subsystem: ASUSTeK Computer Inc. Device [1043:8694]
Flags: bus master, medium devsel, latency 0, IRQ 171
Memory at 2fff010000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Kernel driver in use: vfio-pci
Kernel modules: xhci_pci

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1) (prog-if 00 [VGA controller])
Subsystem: eVga.com. Corp. Device [3842:4995]
Flags: bus master, fast devsel, latency 0, IRQ 170
Memory at db000000 (32-bit, non-prefetchable) [size=16M]
Memory at 90000000 (64-bit, prefetchable) [size=256M]
Memory at a0000000 (64-bit, prefetchable) [size=32M]
I/O ports at e000 [size=128]
Expansion ROM at dc000000 [disabled] [size=512K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
Kernel driver in use: vfio-pci
Kernel modules: nouveau

01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)
Subsystem: eVga.com. Corp. Device [3842:4995]
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at dc080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

Kernel CLI:

initrd=\initramfs-linux-vfio.img cryptdevice=UUID=bc838115-107d-41ef-a076-6a59a2a6bb43:cryptroot root=/dev/mapper/cryptroot pcie_acs_override=downstream intel_iommu=on rw

IOMMU status:

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] ACPI: DMAR 0x00000000365D85A8 0000B8 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.074566] DMAR: Host address width 39
[    0.074568] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.074573] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.074574] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.074577] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.074577] DMAR: RMRR base: 0x0000003630a000 end: 0x00000036329fff
[    0.074578] DMAR: RMRR base: 0x00000038800000 end: 0x0000003affffff
[    0.074580] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.074581] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.074581] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.074582] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.075960] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.683801] DMAR: [Firmware Bug]: RMRR entry for device 04:00.0 is broken - applying workaround
[    0.683804] DMAR: [Firmware Bug]: RMRR entry for device 0b:00.0 is broken - applying workaround
[    0.683806] DMAR: No ATSR found
[    0.683862] DMAR: dmar0: Using Queued invalidation
[    0.684069] DMAR: dmar1: Using Queued invalidation
[    0.684073] DMAR: Setting RMRR:
[    0.684099] DMAR: Setting identity map for device 0000:00:02.0 [0x38800000 - 0x3affffff]
[    0.684107] DMAR: Setting identity map for device 0000:00:14.0 [0x3630a000 - 0x36329fff]
[    0.684115] DMAR: Setting identity map for device 0000:04:00.0 [0x3630a000 - 0x36329fff]
[    0.684121] DMAR: Setting identity map for device 0000:0b:00.0 [0x3630a000 - 0x36329fff]
[    0.684125] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.684128] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.684155] DMAR: Intel(R) Virtualization Technology for Directed I/O
[   88.848284] AMD IOMMUv2 driver by Joerg Roedel <jroedel suse de>
[   88.848285] AMD IOMMUv2 functionality not available on this system
[111338.012019] DMAR: DRHD: handling fault status reg 2
[111338.012025] DMAR: DMAR:[DMA Read] Request device [00:14.0] fault addr 12acc0000 
                DMAR:[fault reason 02] Present bit in context entry is clear

IOMMU groups:

IOMMU group 0
00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:191f] (rev 07)
IOMMU group 1
00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU group 2
00:01.1 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x8) [8086:1905] (rev 07)
IOMMU group 3
00:02.0 VGA compatible controller [0300]: Intel Corporation Skylake Integrated Graphics [8086:1912] (rev 06)
IOMMU group 4
00:14.0 USB controller [0c03]: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f] (rev 31)
IOMMU group 5
00:16.0 Communication controller [0780]: Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a] (rev 31)
IOMMU group 6
00:17.0 SATA controller [0106]: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] [8086:a102] (rev 31)
IOMMU group 7
00:1b.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Root Port #17 [8086:a167] (rev f1)
IOMMU group 8
00:1c.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #1 [8086:a110] (rev f1)
00:1c.2 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #3 [8086:a112] (rev f1)
00:1c.4 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #5 [8086:a114] (rev f1)
04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
05:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
06:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
06:02.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
06:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
06:04.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
06:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
06:06.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
06:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1187]
07:00.0 USB controller [0c03]: VIA Technologies, Inc. VL805 USB 3.0 Host Controller [1106:3483] (rev 01)
08:00.0 Network controller [0280]: Broadcom Corporation BCM4360 802.11ac Wireless Network Adapter [14e4:43a0] (rev 03)
09:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
0b:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
0e:00.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
0f:00.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
0f:01.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
0f:02.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
0f:04.0 PCI bridge [0604]: Intel Corporation DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015] [8086:1578]
44:00.0 USB controller [0c03]: Intel Corporation DSL6540 USB 3.1 Controller [Alpine Ridge] [8086:15b6]
IOMMU group 9
00:1d.0 PCI bridge [0604]: Intel Corporation Sunrise Point-H PCI Express Root Port #9 [8086:a118] (rev f1)
79:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller [144d:a802] (rev 01)
IOMMU group 10
00:1f.0 ISA bridge [0601]: Intel Corporation Sunrise Point-H LPC Controller [8086:a145] (rev 31)
00:1f.2 Memory controller [0580]: Intel Corporation Sunrise Point-H PMC [8086:a121] (rev 31)
00:1f.3 Audio device [0403]: Intel Corporation Sunrise Point-H HD Audio [8086:a170] (rev 31)
00:1f.4 SMBus [0c05]: Intel Corporation Sunrise Point-H SMBus [8086:a123] (rev 31)
IOMMU group 11
00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-V [8086:15b8] (rev 31)
IOMMU group 12
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb0] (rev a1)
IOMMU group 13
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450] [1002:677b]
02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series] [1002:aa98]

Hardware:

H/W path                   Device      Class          Description
=================================================================
                                       system         System Product Name (SKU)
/0                                     bus            MAXIMUS VIII EXTREME
/0/0                                   memory         64KiB BIOS
/0/42                                  memory         128KiB L1 cache
/0/43                                  memory         128KiB L1 cache
/0/44                                  memory         1MiB L2 cache
/0/45                                  memory         8MiB L3 cache
/0/46                                  processor      Core i7 (To Be Filled By O.E.M.)
/0/47                                  memory         32GiB System Memory
/0/47/0                                memory         16GiB DIMM Synchronous 2133 MHz (0.5 ns)
/0/47/1                                memory         [empty]
/0/47/2                                memory         16GiB DIMM Synchronous 2133 MHz (0.5 ns)
/0/47/3                                memory         [empty]
/0/100                                 bridge         Skylake Host Bridge/DRAM Registers
/0/100/1                               bridge         Skylake PCIe Controller (x16)
/0/100/1/0                             display        GM200 [GeForce GTX 980 Ti]
/0/100/1/0.1                           multimedia     NVIDIA Corporation
/0/100/1.1                             bridge         Skylake PCIe Controller (x8)
/0/100/1.1/0                           display        Caicos PRO [Radeon HD 7450]
/0/100/1.1/0.1                         multimedia     Caicos HDMI Audio [Radeon HD 6400 Series]
/0/100/2                               display        Skylake Integrated Graphics
/0/100/14                              bus            Sunrise Point-H USB 3.0 xHCI Controller
/0/100/16                              communication  Sunrise Point-H CSME HECI #1
/0/100/17                              storage        Sunrise Point-H SATA controller [AHCI mode]
/0/100/1b                              bridge         Sunrise Point-H PCI Root Port #17
/0/100/1c                              bridge         Sunrise Point-H PCI Express Root Port #1
/0/100/1c/0                            bus            ASM1142 USB 3.1 Host Controller
/0/100/1c/0/0              usb2        bus            xHCI Host Controller
/0/100/1c/0/0/1                        storage        CDDVDW SU-208DB
/0/100/1c/0/1              usb1        bus            xHCI Host Controller
/0/100/1c/0/1/2                        multimedia     HD Pro Webcam C920
/0/100/1c.2                            bridge         Sunrise Point-H PCI Express Root Port #3
/0/100/1c.2/0                          bridge         ASMedia Technology Inc.
/0/100/1c.2/0/1                        bridge         ASMedia Technology Inc.
/0/100/1c.2/0/1/0                      bus            VL805 USB 3.0 Host Controller
/0/100/1c.2/0/1/0/0        usb4        bus            xHCI Host Controller
/0/100/1c.2/0/1/0/1        usb3        bus            xHCI Host Controller
/0/100/1c.2/0/1/0/1/1                  bus            USB2.0 Hub
/0/100/1c.2/0/1/0/1/1/4                generic        FT240X USB FIFO
/0/100/1c.2/0/2                        bridge         ASMedia Technology Inc.
/0/100/1c.2/0/2/0                      network        BCM4360 802.11ac Wireless Network Adapter
/0/100/1c.2/0/3                        bridge         ASMedia Technology Inc.
/0/100/1c.2/0/3/0                      storage        ASM1062 Serial ATA Controller
/0/100/1c.2/0/4                        bridge         ASMedia Technology Inc.
/0/100/1c.2/0/5                        bridge         ASMedia Technology Inc.
/0/100/1c.2/0/5/0                      bus            ASM1042A USB 3.0 Host Controller
/0/100/1c.2/0/5/0/0        usb6        bus            xHCI Host Controller
/0/100/1c.2/0/5/0/0/2                  bus            USB3.0 Hub
/0/100/1c.2/0/5/0/0/2/4                bus            USB3.0 Hub
/0/100/1c.2/0/5/0/1        usb5        bus            xHCI Host Controller
/0/100/1c.2/0/5/0/1/1                  generic        USB 10/100 LAN
/0/100/1c.2/0/5/0/1/2                  bus            USB2.0 Hub
/0/100/1c.2/0/5/0/1/2/1                bus            USB2.0 Hub
/0/100/1c.2/0/5/0/1/2/4                bus            USB2.0 Hub
/0/100/1c.2/0/5/0/1/2/4/4              input          Yubikey 4 OTP+U2F+CCID
/0/100/1c.2/0/6                        bridge         ASMedia Technology Inc.
/0/100/1c.2/0/7                        bridge         ASMedia Technology Inc.
/0/100/1c.4                            bridge         Sunrise Point-H PCI Express Root Port #5
/0/100/1c.4/0                          bridge         DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
/0/100/1c.4/0/0                        bridge         DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
/0/100/1c.4/0/1                        bridge         DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
/0/100/1c.4/0/2                        bridge         DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
/0/100/1c.4/0/2/0                      bus            DSL6540 USB 3.1 Controller [Alpine Ridge]
/0/100/1c.4/0/2/0/0        usb8        bus            xHCI Host Controller
/0/100/1c.4/0/2/0/1        usb7        bus            xHCI Host Controller
/0/100/1c.4/0/2/0/1/2                  bus            USB 2.0 Hub
/0/100/1c.4/0/2/0/1/2/2                input          CST Laser Trackball
/0/100/1c.4/0/2/0/1/2/3                input          ErgoDox ergonomic keyboard
/0/100/1c.4/0/4                        bridge         DSL6540 Thunderbolt 3 Bridge [Alpine Ridge 4C 2015]
/0/100/1d                              bridge         Sunrise Point-H PCI Express Root Port #9
/0/100/1d/0                            storage        NVMe SSD Controller
/0/100/1f                              bridge         Sunrise Point-H LPC Controller
/0/100/1f.2                            memory         Memory controller
/0/100/1f.3                            multimedia     Sunrise Point-H HD Audio
/0/100/1f.4                            bus            Sunrise Point-H SMBus
/0/100/1f.6                enp0s31f6   network        Ethernet Connection (2) I219-V
/1                                     power          To Be Filled By O.E.M.
/2                         scsi8       storage        
/3                         virbr0-nic  network        Ethernet interface
/4                         enp11s0u1   network        Ethernet interface

Qemu CLI:

QEMU_AUDIO_DRV=pa \
QEMU_PA_SAMPLES=4096 \
QEMU_PA_SERVER=localhost \
PULSE_SERVER=localhost  \
qemu-system-x86_64 \
-enable-kvm \
-machine type=q35,accel=kvm \
-cpu host,kvm=off \
-smp 4,sockets=1,cores=2,threads=2 \
-m 8G \
-mem-path /dev/hugepages \
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/sda,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/tmp.CpwZ4KB6PL \
-soundhw hda \
-usb -device vfio-pci,host=00:14.0 \
-rtc clock=host,base=utc

--
Lance R. Vick

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]