[vfio-users] RLIMIT_MEMLOCK exceeded when enabling intremap

[Prasun Ratn]

Hi

I am adding an IOMMU device using the following libvirt syntax (taken
from https://libvirt.org/formatdomain.html#elementsIommu)

    <iommu model='intel'>
      <driver intremap='on'/>
    </iommu>

When I try to start the VM, it fails. If I remove the above 3 lines it
starts fine.

    error: Failed to start domain rhel7.3-32T-nvme-ich9
    error: internal error: qemu unexpectedly closed the monitor:
2018-06-28T15:24:31.401831Z qemu-kvm: -device
vfio-pci,host=82:00.0,id=hostdev1,bus=pcie.0,addr=0xa: VFIO_MAP_DMA:
-12
    2018-06-28T15:24:31.401854Z qemu-kvm: -device
vfio-pci,host=82:00.0,id=hostdev1,bus=pcie.0,addr=0xa:
vfio_dma_map(0x556478bc0820, 0xc0000, 0x7ff40000, 0x7fd94e4c0000) =
-12 (Cannot allocate memory)
    2018-06-28T15:24:31.450793Z qemu-kvm: -device
vfio-pci,host=82:00.0,id=hostdev1,bus=pcie.0,addr=0xa: VFIO_MAP_DMA:
-12
    2018-06-28T15:24:31.450804Z qemu-kvm: -device
vfio-pci,host=82:00.0,id=hostdev1,bus=pcie.0,addr=0xa:
vfio_dma_map(0x556478bc0820, 0x100000000, 0x180000000, 0x7fd9ce400000)
= -12 (Cannot allocate memory)
    2018-06-28T15:24:31.450878Z qemu-kvm: -device
vfio-pci,host=82:00.0,id=hostdev1,bus=pcie.0,addr=0xa: vfio error:
0000:82:00.0: failed to setup container for group 37: memory listener
initialization failed for container: Cannot allocate memory

In dmesg I see this:

    [189435.289113] vfio_pin_pages_remote: RLIMIT_MEMLOCK (9663676416) exceeded
    [189435.338165] vfio_pin_pages_remote: RLIMIT_MEMLOCK (9663676416) exceeded

I have enough free memory (I think) and at the failing point enough
memory seems to be available.

    $ free -h
                  total        used        free      shared
buff/cache   available
    Mem:           125G        1.4G        123G         17M
1.1G        123G
    Swap:          1.0G          0B        1.0G

Here's the ulimit -l output (I changed limits.conf to set memlock to
unlimited for qemu user and qemu group)

    $ ulimit -l
    unlimited


    $ sudo -u qemu sh -c "ulimit -l"
    unlimited

memlock limit using systemctl

    $ systemctl show libvirtd.service | grep LimitMEMLOCK
    LimitMEMLOCK=18446744073709551615

SELinux is disabled

    $ sestatus
    SELinux status:                 disabled


libvirt and kernel version

    $ virsh version
    Compiled against library: libvirt 4.1.0
    Using library: libvirt 4.1.0
    Using API: QEMU 4.1.0
    Running hypervisor: QEMU 2.9.0

    $ uname -r
    3.10.0-693.5.2.el7.x86_64

    $ cat /etc/redhat-release
    Red Hat Enterprise Linux Server release 7.4 (Maipo)

Any idea how to figure out why we are exceeding the memlock limit?

Thanks
Prasun