[libvirt] QEMU/KVM support for non zero PCI domain

[Laine Stump]

On 04/16/2013 11:23 AM, Charles Peri wrote:

>
> Hi,
>   I am trying to run Virt-Manager, using KVM on a SGIUV100 system with RHEL 6.1. I am trying to enable PCI pass-thorugh for the Mellanox, but the PCI domain for the device is 0001. 

>
> > 4) Does libvirt actually get to the point of running qemu or kvm command?
>
> > 5) If the answer to (4) is "yes", can you send the qemu command that is
> > generated (it will be at the end of /var/log/libvirt/qemu/${guestname}.log)
>
>
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none /usr/libexec/qemu-kvm -name UV2-VM1-test1 -S -M rhel6.4.0 -enable-kvm -m 8192 -smp 4,sockets=4,cores=1,threads=1 -uuid e2428c26-f06a-b03c-c8d5-c528f2e85596 -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/UV2-VM1-test1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/var/lib/libvirt/images/UV2-VM1-test1.img,if=none,id=drive-ide0-0-0,format=raw,cache=none -device ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:51:5f:5f,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -vga cirrus -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5

Okay. The important part here is this:

   -pci-assign,host=02:00.4,id=hostdev0,configfd=24,bus=pci.0,addr=0x6


This shows that libvirt is requesting to assign the device in domain 0,
bus 2, slot 0, function 4,

>> 0001:01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)
>> 0001:02:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)
>>
>> When I try to add the device, I get an error stating that "Can't assign device inside non- zero PCI segment, as this KVM module doesn't support 

I don't see this error in the above log, nor how it could have been
generated. There is nothing in the pci-assign device about a non-zero
domain. So I'm not sure how you arrived at that message.


At any rate, I've done some investigation and learned the following:

1) The qemu-0.12.x that is in RHEL6 only supports assigning devices in
domain 0.

2) "At some point" since then, the "host" option has been expanded to
allow including a domain id, e.g. "-device
pci-assign,host=0001:02:00.0,...", but that support was not backported
to RHEL.

3) Even if you were to have a version of qemu that supported assigning a
device in a non-zero domain, libvirt ignores the domain in the device
address, and only tells qemu about bus:slot.function.


So, libvirt should add support for assigning pci devices in non-0
domains (and it's reasonably simple to do so), but it will only work on
newer versions of qemu than what you have, so you would have to build
your own (unsupported) upstream versions of both qemu and libvirt.