[vfio-users] UEFI + PCI USB Device

Brett Foster fosterb at edgeandvertex.org
Sun Nov 1 19:18:24 UTC 2015 

Okaaaaay... I am much more confused now. I haven't tracked down the host
issue I reported, but....

The VM issue appears to be unrelated not to the USB. I removed the USB PCI
card, and could reproduce the 'lock up'. Perhaps it is related to the
radeon card? If I leave windows to boot for like 10 minutes (an accidental
discovery), it will eventually startup. The HDMI audio device reports code
10. The kernel logs are a bit funky too:

[ 3648.842842] virbr0: port 2(vnet0) entered disabled state
[ 3648.844097] device vnet0 left promiscuous mode
[ 3648.844120] virbr0: port 2(vnet0) entered disabled state
[ 3649.270209] irq 17: nobody cared (try booting with the "irqpoll" option)
[ 3649.270212] CPU: 4 PID: 0 Comm: swapper/4 Not tainted
4.0.4-301.fc22.x86_64 #1
[ 3649.270213] Hardware name: ASUS All Series/Z87M-PLUS, BIOS 1107
11/04/2014
[ 3649.270214]  0000000000000000 b3369ae444567343 ffff88063ed03e18
ffffffff81782644
[ 3649.270216]  0000000000000000 ffff880620f0ea8c ffff88063ed03e48
ffffffff810f6e27
[ 3649.270217]  ffffffff81cb8140 ffff880620f0ea00 0000000000000000
0000000000000011
[ 3649.270219] Call Trace:
[ 3649.270220]  <IRQ>  [<ffffffff81782644>] dump_stack+0x45/0x57
[ 3649.270228]  [<ffffffff810f6e27>] __report_bad_irq+0x37/0xd0
[ 3649.270230]  [<ffffffff810f71c8>] note_interrupt+0x258/0x2a0
[ 3649.270232]  [<ffffffff810f4693>] handle_irq_event_percpu+0x133/0x1a0
[ 3649.270233]  [<ffffffff810f473b>] handle_irq_event+0x3b/0x60
[ 3649.270235]  [<ffffffff810f7759>] handle_fasteoi_irq+0x79/0x120
[ 3649.270238]  [<ffffffff81016514>] handle_irq+0x74/0x140
[ 3649.270241]  [<ffffffff810bc57a>] ? atomic_notifier_call_chain+0x1a/0x20
[ 3649.270243]  [<ffffffff8178ba1f>] do_IRQ+0x4f/0xf0
[ 3649.270246]  [<ffffffff8178976d>] common_interrupt+0x6d/0x6d
[ 3649.270246]  <EOI>  [<ffffffff81618b7d>] ? cpuidle_enter_state+0x5d/0x170
[ 3649.270249]  [<ffffffff81618b6d>] ? cpuidle_enter_state+0x4d/0x170
[ 3649.270251]  [<ffffffff81618cc7>] cpuidle_enter+0x17/0x20
[ 3649.270253]  [<ffffffff810df3c4>] cpu_startup_entry+0x314/0x430
[ 3649.270255]  [<ffffffff8111283c>] ?
clockevents_register_device+0xbc/0x130
[ 3649.270258]  [<ffffffff8104a9b5>] start_secondary+0x1a5/0x1f0
[ 3649.270259] handlers:
[ 3649.270267] [<ffffffffa06dbad0>] azx_interrupt [snd_hda_controller]
[ 3649.270267] Disabling IRQ #17


On Sun, Nov 1, 2015 at 10:28 AM, Brett Foster <fosterb at edgeandvertex.org>
wrote:

> Howdy,
>
> I ran a couple more tests and I think I see the problem. It is remarkably
> similar to what I saw on the reset-affected Radeon cards. If i start
> windows and shutdown windows properly - usb works great. If I start windows
> and force stop. It fails. Unlike previous manifestations, this time it
> fails at the windows boot screen. BUT it gets worse...
>
> If the host is restarted (warm reset?), the host will be unable to boot.
> The log spits out some gibberish related to USB then halts. It's not a
> kernel panic. It looks like both windows or linux drivers get stuck in a
> way that leaves the system half alive and half dead. The host responds to
> CTRL-ALT-DELETE, however it will only partially shutdown before getting
> stuck again. The host must be physically powered down (i.e. AC removed)
> before the host will operate properly.
>
> I am tempted to look in to the linux xhci driver to see why it doesn't
> fail gracefully.
>
> Thoughts?
>
> Brett
>
> On Sun, Nov 1, 2015 at 8:15 AM, francesco dicarlo <evilsephiroth at gmail.com
> > wrote:
>
>> Well, I'm intereted in this issue too. Buying an additional pci usb card
>> and don't want to get stuck in the same situation...
>>
>> 2015-11-01 12:35 GMT+01:00 Blank Field <ihatethisfield at gmail.com>:
>>
>>> Reply to the list. Use reply all button.
>>>
>>> I have no idea how to sort out your issue:(
>>> On Nov 1, 2015 8:56 AM, "Brett Foster" <fosterb at edgeandvertex.org>
>>> wrote:
>>>
>>>> Just to follow up -- the comment in the source code is this:
>>>>
>>>> // After Chip Hardware Reset wait until the Controller Not Ready (CNR)
>>>> flag
>>>> // in the USBSTS is '0' before writing any xHC Operational or Runtime
>>>> registers.
>>>>
>>>> I assume the card is getting stuck in a CNR state. Maybe it's another
>>>> one of those improperly implemented reset situations.
>>>>
>>>> On Sat, Oct 31, 2015 at 10:42 PM, Brett Foster <
>>>> fosterb at edgeandvertex.org> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Neat. So I enabled the logging and we get this....
>>>>>
>>>>> XhcCreateUsb3Hc: Capability length 0x20
>>>>> XhcCreateUsb3Hc: HcSParams1 0x8000820
>>>>> XhcCreateUsb3Hc: HcSParams2 0x24000011
>>>>> XhcCreateUsb3Hc: HcCParams 0x14051C7
>>>>> XhcCreateUsb3Hc: DBOff 0x800
>>>>> XhcCreateUsb3Hc: RTSOff 0x600
>>>>> XhcCreateUsb3Hc: UsbLegSupOffset 0x0
>>>>> XhcCreateUsb3Hc: DebugCapSupOffset 0x50
>>>>> XhcSetBiosOwnership: called to set BIOS ownership
>>>>> XhcResetHC!
>>>>> ASSERT
>>>>> /home/jenkins/workspace/edk2/rpmbuild/rpm/BUILD/edk2.git/MdeModulePkg/Bus/Pci/XhciDxe/Xhci.c(2032):
>>>>> !((((BOOLEAN)(((XhcReadOpReg ((Xhc), (0x0004))) & ((0x00000800))) ==
>>>>> ((0x00000800))))))
>>>>>
>>>>> I feel like this issue is going to be much harder to debug than it is
>>>>> worth. Full log is attached.
>>>>>
>>>>>
>>>>> On Fri, Oct 30, 2015 at 11:42 PM, Blank Field <
>>>>> ihatethisfield at gmail.com> wrote:
>>>>>
>>>>>> Add isa debug console to your script:
>>>>>> -global isa-debugcon.iobase=0x402 \
>>>>>> -debugcon file:"debug.log"
>>>>>> OVMF will print out verbose debug messages to $(pwd)/debug.log
>>>>>> On Oct 31, 2015 6:27 AM, "Brett Foster" <fosterb at edgeandvertex.org>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> So I've been fumbling my way through setting up a virtual machine
>>>>>>> with GPU pass through. I was able to get UEFI + ATI R5 230 to windows 8.1
>>>>>>> installer. Great... now I wanted to add a PCI USB 3.0 card to the VM. When
>>>>>>> I do, the VM locks up at the main OVMF screen that says "TianoCore" and so
>>>>>>> many bytes of system memory tested.
>>>>>>>
>>>>>>> Unlike the GPU I found I couldn't figure out great way to prevent
>>>>>>> the standard linux drivers from picking up the device before being able to
>>>>>>> map to vfio-pci. So I unbind the device when booting:
>>>>>>>
>>>>>>> echo 0000:04:00.0 > /sys/bus/pci/drivers/xhci_hcd/unbind
>>>>>>>
>>>>>>> Thoughts?
>>>>>>>
>>>>>>> Kernel: 4.0.4-301.fc22.x86_64
>>>>>>> Qemu: qemu-2.3.0-3.fc22
>>>>>>>
>>>>>>> The only seemingly relevant messages in the kernel log are:
>>>>>>> [ 6209.596712] vfio-pci 0000:05:00.1: enabling device (0400 -> 0402)
>>>>>>> [ 6209.619946] vfio-pci 0000:04:00.0: enabling device (0400 -> 0402)
>>>>>>>
>>>>>>> The card is on its own group:
>>>>>>> /sys/kernel/iommu_groups/13/devices/0000:04:00.005:00.0 VGA
>>>>>>> compatible controller [0300]:
>>>>>>> The GPU is on these groups:
>>>>>>> /sys/kernel/iommu_groups/14/devices/0000:05:00.0
>>>>>>> /sys/kernel/iommu_groups/14/devices/0000:05:00.1
>>>>>>>
>>>>>>> 04:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201
>>>>>>> USB 3.0 Host Controller [1912:0014] (rev 03) (prog-if 30 [XHCI])
>>>>>>>     Subsystem: Renesas Technology Corp. uPD720201 USB 3.0 Host
>>>>>>> Controller [1912:0014]
>>>>>>>     Flags: bus master, fast devsel, latency 0, IRQ 19
>>>>>>>     Memory at f7200000 (64-bit, non-prefetchable) [size=8K]
>>>>>>>     Capabilities: <access denied>
>>>>>>>     Kernel driver in use: vfio-pci
>>>>>>> Advanced Micro Devices, Inc. [AMD/ATI] Caicos PRO [Radeon HD 7450]
>>>>>>> [1002:677b] (prog-if 00 [VGA controller])
>>>>>>>     Subsystem: ASUSTeK Computer Inc. Device [1043:3025]
>>>>>>>     Flags: bus master, fast devsel, latency 0, IRQ 16
>>>>>>>     Memory at e0000000 (64-bit, prefetchable) [size=256M]
>>>>>>>     Memory at f7120000 (64-bit, non-prefetchable) [size=128K]
>>>>>>>     I/O ports at c000 [size=256]
>>>>>>>     Expansion ROM at f7100000 [disabled] [size=128K]
>>>>>>>     Capabilities: <access denied>
>>>>>>>     Kernel driver in use: vfio-pci
>>>>>>>     Kernel modules: radeon
>>>>>>>
>>>>>>> 05:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI]
>>>>>>> Caicos HDMI Audio [Radeon HD 6400 Series] [1002:aa98]
>>>>>>>     Subsystem: ASUSTeK Computer Inc. Device [1043:aa98]
>>>>>>>     Flags: bus master, fast devsel, latency 0, IRQ 17
>>>>>>>     Memory at f7140000 (64-bit, non-prefetchable) [size=16K]
>>>>>>>     Capabilities: <access denied>
>>>>>>>     Kernel driver in use: vfio-pci
>>>>>>>     Kernel modules: snd_hda_intel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> vfio-users mailing list
>>>>>>> vfio-users at redhat.com
>>>>>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>>>>>
>>>>>>>
>>>>>
>>>>
>>> _______________________________________________
>>> vfio-users mailing list
>>> vfio-users at redhat.com
>>> https://www.redhat.com/mailman/listinfo/vfio-users
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20151101/0694f83a/attachment.htm>

More information about the vfio-users mailing list