[vfio-users] Application examples using vfio ?

[Alex Williamson]

On Tue, 6 Feb 2018 19:27:52 +0200
Ran Shalit <ranshalit at gmail.com> wrote:

> On Tue, Feb 6, 2018 at 5:41 PM, Alex Williamson
> <alex.williamson at redhat.com> wrote:
> > On Tue, 6 Feb 2018 17:20:08 +0200
> > Ran Shalit <ranshalit at gmail.com> wrote:
> >  
> >> On Tue, Feb 6, 2018 at 4:56 PM, Alex Williamson
> >> <alex.williamson at redhat.com> wrote:  
> >> > On Tue, 6 Feb 2018 16:24:06 +0200
> >> > Ran Shalit <ranshalit at gmail.com> wrote:
> >> >  
> >> >> Hello,
> >> >>
> >> >> I need to write vfio userspace driver for pci express.
> >> >> I've been searching for some examples of using vfio, but did not find any,
> >> >> Is there a git/examples of userspace application using vfio ?  
> >> >
> >> > git://git.qemu.org/qemu.git
> >> > git://dpdk.org/dpdk
> >> > https://github.com/andre-richter/rVFIO
> >> > https://github.com/MicronSSD/unvme
> >> > https://github.com/awilliam/tests  
> >>
> >> Thank you.
> >> I am trying to understand how to use it with dma.
> >> I've been checking now the links, but did not find yet in the above
> >> example how to both map DMA  memory and read/write from that memory.
> >> I see that the above test git contains tests which map/unmap DMA, but
> >> not reading/writing from that dma memory afterwards (there is a
> >> seperate test which read/write from memory but without dma).
> >> Is that correct or did I miss something ? Is there any example which
> >> shows how to both map and then read/write ?  
> >
> > Initiating DMA is device specific, you need to understand the
> > programming model of the device for that.  
> 
> I haven't yet found any example, so I please want to ask one more on dma issue:
> As far as I know, in kernel drivers, you can do the following:
> 1. allocate dma memory
> 2. program the device to be familiar with the dma address,
> 3. and then for write:  trigger a dma transaction by writing in BARs
> specific register.
> 4. and for read:  usually just waiting for interrupt
> So, if I understand correctly in the dma test example , only (1) above
> is implemented, and the rest are not implemented becuase they are
> device specific (and the test are generic), Right ?

Correct.
 
> I would also like to ask about interrupt, I read the documentation in
> vfio.h, but I can't find how the irq is actually set.
> I don't see that VFIO_DEVICE_SET_IRQS
> has parameter for irq number. Does it ?

The VFIO_DEVICE_SET_IRQS ioctl takes the following structure:

struct vfio_irq_set {
        __u32   argsz;
        __u32   flags;
#define VFIO_IRQ_SET_DATA_NONE          (1 << 0) /* Data not present */
#define VFIO_IRQ_SET_DATA_BOOL          (1 << 1) /* Data is bool (u8) */
#define VFIO_IRQ_SET_DATA_EVENTFD       (1 << 2) /* Data is eventfd (s32) */
#define VFIO_IRQ_SET_ACTION_MASK        (1 << 3) /* Mask interrupt */
#define VFIO_IRQ_SET_ACTION_UNMASK      (1 << 4) /* Unmask interrupt */
#define VFIO_IRQ_SET_ACTION_TRIGGER     (1 << 5) /* Trigger interrupt */
        __u32   index;
        __u32   start;
        __u32   count;
        __u8    data[];
};

@index is the basis for the interrupt being configured, where vfio-pci
defines the following indexes:

enum {
        VFIO_PCI_INTX_IRQ_INDEX,
        VFIO_PCI_MSI_IRQ_INDEX,
        VFIO_PCI_MSIX_IRQ_INDEX,
        VFIO_PCI_ERR_IRQ_INDEX,
        VFIO_PCI_REQ_IRQ_INDEX,
        VFIO_PCI_NUM_IRQS
};

The first 3 are the normal PCI interrupt types.  @start and @count can
be used to manipulate multiple sub-indexes, or vectors, at once.  For
instance:

  .flags = VFIO_IRQ_SET_DATA_EVENTFD | VFIO_IRQ_SET_ACTION_TRIGGER,
  .index = VFIO_PCI_MSI_IRQ_INDEX,
  .start = 0,
  .count = 1,

and providing an eventfd for signaling in the data section would enable
MSI with 1 vector and trigger the provided eventfd when it occurs.
There's of course an IRQ info ioctl for each index to probe the device
capabilities.  Interrupts are not configured by direct manipulation of
the interrupt related registers in PCI config or MMIO space for
vfio-pci.  Thanks,

Alex