[vfio-users] [help] 2 identical GPUs in Arch

[Alex Williamson]

On Fri, 19 Feb 2016 16:43:47 -0500
Ben J <btpprograms at gmail.com> wrote:

> For your latter question, it doesn't seem like you should need to modify it
> as his original script already generically finds all GPUs. Did it not work
> when you tried it? Also, what type of cards are you using? If they're
> NVIDIA with nvidia drivers then I doubt that script will work for you.
> I'm using the following guide:
> https://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html.
> I'm having trouble adapting some of the Fedora focused instructions for use
> in Arch.
> 
> *1. Dracut*
> Dracut isn't used in Arch but the mkinitcpio is. You're asked to
> insert *add_drivers+="vfio
> vfio_iommu_type1 vfio_pci vfio_virqfd" *in: */etc/dracut.conf.d/local.conf*.
> Farther down, I'm supposed to insert
> *install_items+="/sbin/vfio-pci-override-vga.sh
> /usr/bin/find /usr/bin/dirname" *in: */etc/dracut.conf.d/local.conf*.
> 
> Should I instead add these lines to a modprobe.d conf file or should I add
> it to the modules line in */etc/mkinitcpio.conf*?
> 
> *2. driver_override*
> My bash-foo isn't very good and I need to adapt the following script (see
> below) so that it points to the correct directories. In Arch, the directory
> for the GPU (02:00.0) and GPU audio (02:00.1) are here:
> */sys/devices/pci0000:00/0000:00:02.0/0000:02:00.0/* &
> */sys/devices/pci0000:00/0000:00:02.0/0000:02:00.1/*
> 
> 
> 
> 
> 
> 
> 
> *    #!/bin/sh    DEVS="0000:02:00.0 0000:02:00.1"    for DEV in $DEVS;
> do echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override    done
> modprobe -i vfio-pci*

There are really just so many ways to solve this problem, don't feel
like you need to stick with the approaches I've given.  If you want
something that only depends on modprobe.d, then the "install" directive
is really powerful.  For instance, let's say I have a pair of identical
nvidia cards, one at 01:00.0 and the other at 02:00.0 and I want to use
02: for my guest.  I could do something like:

install nvidia "echo vfio-pci > /sys/bus/pci/devices/0000:02:00.0/driver_override; modprobe --ignore-install nvidia"

This tells modprobe that before loading the nvidia driver, set the
driver override so that only vfio-pci can claim 02:00.0, then load the
nvidia driver.  You can do the same with the audio function and
snd_hda_intel driver.  You can wait til later to actually load the
vfio-pci module, it's not really necessary to include them in your
initramfs.

As for sysfs paths, the actual devices do live in /sys/devices/, but
there are links to them in /sys/bus/pci/devices/, this is a kernel
property, your choice of distribution isn't going to change that.
Thanks,

Alex