[libvirt-users] intel quad gigabit nic and pci passthrough

Sun Feb 2 11:16:25 UTC 2014

Laine,

thanks for your reply.  Following your suggestion to check the last few lines of the domain logs, I found this snippet:

2014-01-14 00:08:52.281+0000: 27653: error : virPCIDeviceIsAssignable:2223 : internal error: Device 0000:06:00.1 is behind a switch lacking ACS and cannot be assigned

Google told me a very simple solution: enable relaxed_acs_check = 1 in qemu.conf.

Now it works as expected.

Thanks,
IvanK.

----- Original Message -----
From: "Laine Stump" <laine at laine.org>
To: libvirt-users at redhat.com
Cc: chepati at yahoo.com
Sent: Tuesday, January 28, 2014 12:58:05 PM
Subject: Re: [libvirt-users] intel quad gigabit nic and pci passthrough

On 01/25/2014 06:54 PM, Ivan Kabaivanov wrote:
> Hi all
>
> I have a very weird case of pci passthrough.
>
> I have a machine with 7 network interfaces, all of them intel.  Four of them are on one quad giga ethernet device.  If I manually unbind the devices and allow qemu to use them, with intel IOMMU working, everything works like a charm.  Here's how I do it manually:
>
> root at kybrat (x86_64) ~]$ lspci -nn | grep net
> 00:19.0 Ethernet controller [0200]: Intel Corporation 82579V Gigabit Network Connection [8086:1503] (rev 06)
> 02:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06)
> 05:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
> 05:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
> 06:00.0 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
> 06:00.1 Ethernet controller [0200]: Intel Corporation 82571EB Gigabit Ethernet Controller [8086:10a4] (rev 06)
> 0c:00.0 Ethernet controller [0200]: Intel Corporation 82572EI Gigabit Ethernet Controller (Copper) [8086:107d] (rev 06)
>
>
> echo "8086 10a4" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "8086 107d" > /sys/bus/pci/drivers/pci-stub/new_id
> echo "0000:05:00.0" > /sys/bus/pci/devices/0000\:05\:00.0/driver/unbind
> echo "0000:05:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:05:00.1" > /sys/bus/pci/devices/0000\:05\:00.1/driver/unbind
> echo "0000:05:00.1" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:06:00.0" > /sys/bus/pci/devices/0000\:06\:00.0/driver/unbind
> echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:06:00.1" > /sys/bus/pci/devices/0000\:06\:00.1/driver/unbind
> echo "0000:06:00.1" > /sys/bus/pci/drivers/pci-stub/bind
> echo "0000:0c:00.0" > /sys/bus/pci/devices/0000\:0c\:00.0/driver/unbind
> echo "0000:0c:00.0" > /sys/bus/pci/drivers/pci-stub/bind
>
> qemu-system-i386 -enable-kvm \
>                  -m 2048 \
>                  -cpu host \
>                  -machine pc-q35-1.6 \                                           
>                  -drive file=/media/virtual/krym.qcow2,if=virtio \       
>                  -net none \                                                     
>                  -device pci-assign,host=06:00.1,rombar=0 \                      
>                  -device pci-assign,host=05:00.0,rombar=0 \                      
>                  -device pci-assign,host=06:00.0,rombar=0 \                      
>                  -device pci-assign,host=05:00.1,rombar=0 \                      
>                  -device pci-assign,host=0c:00.0,rombar=0 \                      
>                  -cdrom /media/virtual/asg-9.107-33.1.iso \                      
>                  -boot c \                                                       
>                  -vnc :0 \                                             
>                  -runas virtual \                                                
>                  -monitor unix:/media/virtual/control/krym/monitor,server,nowait \                                                                                                               
>                  -rtc base=localtime,clock=host,driftfix=none
>
> This starts qemu and pci passthrough works nicely.
>
> Now, if I try to do the same thing through libvirt, virt-install doesn't like the quad ethernet device pci passthrough and I get the error below:
>
> virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_06_00_1 --host-device=pci_0000_05_00_0 --host-device=pci_0000_06_00_0 --host-device=pci_0000_05_00_1 --host-device=pci_0000_0c_00_0
>
> Starting install...
> ERROR    Requested operation is not valid: PCI device 0000:06:00.1 is not assignable
> Domain installation does not appear to have been successful.
> If it was, you can restart your domain by running:
>   virsh --connect qemu:///system start krym
> otherwise, please restart your installation.

More information is needed to make a guess at the source of the problem:

1) What distro/version of Linux are you running.
2) What version of libvirt?
3) After the failed virt-install command, if the new domain is still
defined, can you provide the output of:

    virsh dumpxml $domain

(if virt-install is automatically undefining the failed domain, then you
can get the xml for the defined domain by appending "--print-xml" to the
virt-install commandline)

4) Also, send the last several lines of
/var/log/libvirt/qemu/$domain.log, in particular the qemu commandline
that was executed to start the domain, and any subsequent error messages.

(Just one possibility - since you are successfully using the q35
machinetype, it's highly likely that your OS is new enough that it
supports vfio for device passthrough; vfio is more restrictive about
what devices can be assigned to a guest, sometimes requiring seemingly
unrelated devices to be assigned to the same guest, and the most recent
libvirts will prefer vfio over legacy kvm device assignment when
available. So it's possible that your manual running of qemu-kvm works
because you are forcing use of legacy kvm assignment, and when libvirt
attempts to use vfio it is failing. This will be made apparent by the
information I requested above.

>
>
> If I try to use just one NIC (any NIC, as long as it's not the quad NIC), things do work:
>
> virt-install --name=krym --ram=2048 --vcpus=1 --cpu=host --import --os-type=linux --disk path=/media/virtual/krym.qcow2,format=qcow2,bus=virtio --nonetworks --graphics vnc,port=5900 --virt-type=kvm --machine=q35 --accelerate --host-device=pci_0000_0c_00_0
>
> Starting install...
> Creating domain...                                                                               |    0 B  00:00:01     
> Cannot open display: 
> Run 'virt-viewer --help' to see a full list of available command line options
> Domain creation completed. You can restart your domain by running:
>   virsh --connect qemu:///system start krym
>
>
>
> So I guess libvirt has some problem with the intel quad NIC.
>
> I will appreciate any help.
>
> Thanks,
> IvanK.
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users at redhat.com
> https://www.redhat.com/mailman/listinfo/libvirt-users
>
>