[vfio-users] Application examples using vfio ?

[Alex Williamson]

On Tue, 6 Feb 2018 17:20:08 +0200
Ran Shalit <ranshalit at gmail.com> wrote:

> On Tue, Feb 6, 2018 at 4:56 PM, Alex Williamson
> <alex.williamson at redhat.com> wrote:
> > On Tue, 6 Feb 2018 16:24:06 +0200
> > Ran Shalit <ranshalit at gmail.com> wrote:
> >  
> >> Hello,
> >>
> >> I need to write vfio userspace driver for pci express.
> >> I've been searching for some examples of using vfio, but did not find any,
> >> Is there a git/examples of userspace application using vfio ?  
> >
> > git://git.qemu.org/qemu.git
> > git://dpdk.org/dpdk
> > https://github.com/andre-richter/rVFIO
> > https://github.com/MicronSSD/unvme
> > https://github.com/awilliam/tests  
> 
> Thank you.
> I am trying to understand how to use it with dma.
> I've been checking now the links, but did not find yet in the above
> example how to both map DMA  memory and read/write from that memory.
> I see that the above test git contains tests which map/unmap DMA, but
> not reading/writing from that dma memory afterwards (there is a
> seperate test which read/write from memory but without dma).
> Is that correct or did I miss something ? Is there any example which
> shows how to both map and then read/write ?

Initiating DMA is device specific, you need to understand the
programming model of the device for that.  The vfio DMA ioctls only map
the buffers through the IOMMU to allow that DMA, they don't control the
device.  The unit tests in the last link doesn't have any device
specific code, they're only performing the mapping without actually
initiating device DMA.  The QEMU examples also mostly configure DMA but
rely on native drivers within the guest VM to program the device to
initiate DMA.  The rVFIO project is also just a wrapper to make vfio
more accessible to scripting languages, so I don't expect actual device
driver code there.  DPDK and the NVME driver should have actual driver
code for the device though.  In fact there's also an NVME project
within QEMU that makes use of vfio that might have driver code.  The
simple logic though is that the DMA map ioctl takes a user provided
virtual buffer and maps it through the IOMMU at the user provided IOVA
(I/O virtual address).  It's up to the user to make the device perform
DMA using that IOVA.  Note that the IOMMU address width is typically
less than the user virtual address space, so identity mapping the IOVA
to the virtual address is not a good strategy.
 
> Another question, just to be sure I'm on the right direction. I wanted
> to write pci express userspace driver (using real device, and not
> related to virtualization). Is vfio a proper solution for this kind of
> thing ?

Yes, that's exactly what DPDK does.

> I ask because there are no plenty of examples as I first expected, and
> I try to understand why there are no many, (probably license issues ?)

vfio is of course part of the kernel and GPL, but userspace drivers are
not necessarily under the same license, so it is possible that the
drivers themselves are more private.  DPDK supports several NICs and
I'd guess the NVME driver handles the standard NVME programming model
and could therefore support devices from many vendors.  Thanks,

Alex