[vfio-users] gpus are still within the same iommu group, kernel 4.4.0, i7 6700

[lucas at herich.net]

Hello everyone,

I am running a fresh Ubuntu 16.04 and want to use kvm/qemu to run a 
windows VM with a gpu passthrough, however after patching the kernel 
both gpus are still within the same iommu group 1.
My system:
- Intel Core i7 6700
- Gigabyte GA-Z170X-UD5 TH Intel Z170
- 1x GeForce GTX 650 Ti (used by the host)
- 1x GeForce GTX 960 (used by the vm)

I am not so well versed with linux, so I read lots of tutorials and 
howtos. However I am kind of stuck right now and do not even know how to 
verify if I used the correct patches (Alex commented within the first 
patch: "This doesn't fix what lspci shows.").
Hopefully someone could give me an advice how to proceed from here or if 
I should give up.

That is what I have done:
* install Ubuntu 16.04 (kernel version 4.4.0)
* install kvm, quemu and so on
* let vfio-pci grab the GTX 960 (see [1] and [2])
* install nvidia driver for the GTX 650 Ti
* build patched kernel (see [3])
* start virtual machine manager
* create a vm with windows 7
* adding passthrough gpu from slot 2 results in (details see [4]): vfio: 
error, group 1 is not viable, please ensure all devices within the 
iommu_group are bound to their vfio bus driver

Thanks,
Lucas


### 1
/etc/modprobe.d/vfio.conf
10de:0fba: GTX 960
10de:1401: GTX 960 audio
```
options vfio-pci ids=10de:0fba,10de:1401
```


### 2
output of: lspci -nnk
http://pastebin.com/KqgSwMQf
```
[...]
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 
[GeForce GTX 650 Ti] [10de:11c6] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. GK106 [GeForce GTX 650 Ti] 
[19da:4263]
	Kernel driver in use: nvidia
	Kernel modules: nvidiafb, nouveau, nvidia_340
01:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio 
Controller [10de:0e0b] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. GK106 HDMI Audio Controller 
[19da:4263]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM206 
[GeForce GTX 960] [10de:1401] (rev a1)
	Subsystem: NVIDIA Corporation GM206 [GeForce GTX 960] [10de:1401]
	Kernel driver in use: vfio-pci
	Kernel modules: nvidiafb, nouveau, nvidia_340
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fba] (rev 
a1)
	Subsystem: NVIDIA Corporation Device [10de:1401]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
[...]
```

output of: dmesg | grep iommu
http://pastebin.com/8TM7mBwy
```
[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-4.4.0-23-generic 
root=/dev/mapper/ubuntu--vg-root ro quiet noplymouth intel_iommu=on 
pcie_acs_override=downstream rd.modules-load=vfio-pci
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-4.4.0-23-generic 
root=/dev/mapper/ubuntu--vg-root ro quiet noplymouth intel_iommu=on 
pcie_acs_override=downstream rd.modules-load=vfio-pci
[    0.615463] iommu: Adding device 0000:00:00.0 to group 0
[    0.615474] iommu: Adding device 0000:00:01.0 to group 1
[    0.615480] iommu: Adding device 0000:00:01.1 to group 1
[...]
[    0.615592] iommu: Adding device 0000:01:00.0 to group 1
[    0.615595] iommu: Adding device 0000:01:00.1 to group 1
[    0.615598] iommu: Adding device 0000:02:00.0 to group 1
[    0.615601] iommu: Adding device 0000:02:00.1 to group 1
```


### 3
/etc/default/grub
(and after saving update-grub)
```
[...]
GRUB_CMDLINE_LINUX_DEFAULT="quiet noplymouth intel_iommu=on 
pcie_acs_override=downstream rd.modules-load=vfio-pci"
[...]
```

patched kernel with ACS quirk
```
git clone git://kernel.ubuntu.com/ubuntu/ubuntu-xenial.git
cd ubuntu-xenial
add +acs to the kernel version in debian.master/changelog
first patch: https://patchwork.ozlabs.org/patch/604533/
second patch: https://patchwork.ozlabs.org/patch/604532/
patch -p1 < first.patch
patch -p1 < second.patch
fakeroot debian/rules clean
fakeroot debian/rules binary-headers binary-generic
reboot
```


### 4
Creating a vm using PCI 0:2:0.0 with the Virtual Machine Manager results 
into:
```
Unable to complete install: 'internal error: process exited while 
connecting to monitor: 2016-06-08T17:00:35.910505Z qemu-system-x86_64: 
-device vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: 
error, group 1 is not viable, please ensure all devices within the 
iommu_group are bound to their vfio bus driver.
2016-06-08T17:00:35.910523Z qemu-system-x86_64: -device 
vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7: vfio: failed to 
get group 1
2016-06-08T17:00:35.910530Z qemu-system-x86_64: -device 
vfio-pci,host=02:00.0,id=hostdev0,bus=pci.0,addr=0x7: Device 
initialization failed'
```