[vfio-users] After CPU pinning with libvirt, performance hasn't improved

[Alex Williamson]

On Sat, Sep 12, 2015 at 9:55 AM, Garrett Powell <
garretttracypowell at gmail.com> wrote:

> Per Alex's recommendation, I've switched from controlling QEMU directly to
> using libvirt. I've been trying for the past few days to get CPU pinning to
> work, but my performance hasn't improved. I should mention that my graphics
> performance is fine. After reading through the entire thread at
> virtualkvm.com, I've found a lot of people with similar issues but no
> solutions that have worked for me. I'm using a single quad-core processor
> with HT.
>
> Here's part of my xml file:
> <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0
> '>
>   . . .
>   <vcpu placement='static'>4</vcpu>
>   <cputune>
>     <vcpupin vcpu='0' cpuset='2'/>
>     <vcpupin vcpu='1' cpuset='3'/>
>     <vcpupin vcpu='2' cpuset='6'/>
>     <vcpupin vcpu='3' cpuset='7'/>
>   </cputune>
>   . . .
>   <cpu mode='host-passthrough'>
>     <topology sockets='1' cores='2' threads='2'/>
>   </cpu>
>   . . .
> </domain>
>
> Here are the core ids for my CPUs (0-7):
> $ cat /proc/cpuinfo | grep "core id"
> core id : 0
> core id : 1
> core id : 2
> core id : 3
> core id : 0
> core id : 1
> core id : 2
> core id : 3
>
> Here are my kernel parameters:
> GRUB_CMDLINE_LINUX_DEFAULT="isolcpus=2,3,6,7 intel_iommu=on
> rd.driver.pre=vfio-pci"
>
> Some things from the forum thread I tried:
>
>    - Using <emulatorpin cpuset='2-3,6-7'/> in the xml file under
>    <cputune> and removing "isolcpus" from my kernel parameters to set cpu
>    affinity
>    - Using "iommu=pt"
>    - Using "nohz=off"
>
>
Pinning the emulator to the same CPUs you're using for the vCPUs seems like
a bad idea.  As I pointed out in my latency post last week, I got the best
latency by pinning the emulator to the opposite thread of the CPU from the
vCPU.  The "iommu=pt" option only affects the DMA performance of host
devices.  The "nohz_full=" option seems to be missing from your list of
things to try.  It also doesn't seem like you're using hugepages.

I run tests where assigning threads to a VM degrades performance versus
isolating the thread and reducing the number of vCPUs in the guest.  More
vCPUs is not always the answer.  Thanks,

Alex
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/vfio-users/attachments/20150912/397a7d2f/attachment.htm>