[vfio-users] A couple notes on DPC latency

[Alex Williamson]

Not worthy of blog post, so I'll record it here...

Playing around with isolcpus=1-3,5-7 and nohz_full=1-3,5-7 on an Intel
quad-core w/ HT, effectively leaving only core0 + its thread pair for
the host (wishing I had a hex core or better)

Without specifying anything special in the xml, the emulator thread(s)
should only run on CPUs 0,4 in this config.

I also tell irqbalance to only route interrupts to CPUs 0,4 with
IRQBALANCE_BANNED_CPUS="EE"

If I expose <cores=3, threads=2> vCPUs to the guest with pinning, DPC
latency looks pretty crappy, spikes up to nearly 20ms (running Win8.1
guest, so I guess the best I can get is 1ms due to DPC/Win8
incompatibility).  Yellow bars as singles or doubles, the rest red.

If instead I expose <core=3, threads=1> and specify <emulatorpin
cpuset='5-7'> my DPC bars are almost entirely yellow with occasional
(rare) spikes to sub-10ms.

If I change only the emulatorpin cpuset from 5-7 to 0,4, I get a *lot*
more red spikes, again reaching nearly 20ms.  This should effectively be
the same as the first test, except with threads=1, isolating just the
emulatorpin change.

So... it doesn't seem like it makes much sense to split the threads of a
core between host and guest (ie. isolcpus=1-3, let 5-7 schedule
normally) since neither the host nor guest schedulers can hope to get
that right.  But, making the emulator run on the idle thread pair seems
to significant improve latency, which may actually prove more worthwhile
than giving it to the VM as a vCPU.

Those of you complaining about stuttering guests or audio may want to
try something similar.  Thanks,

Alex