[libvirt] [V3] RFC for support cache tune in libvirt

[Martin Kletzander]

On Thu, Jan 12, 2017 at 10:58:53AM +0800, 乔立勇(Eli Qiao) wrote:
>2017-01-11 19:09 GMT+08:00 Daniel P. Berrange <berrange at redhat.com>:
>
>> On Wed, Jan 11, 2017 at 11:55:28AM +0100, Martin Kletzander wrote:
>> > On Wed, Jan 11, 2017 at 10:05:26AM +0000, Daniel P. Berrange wrote:
>> > >
>> > > IIUC, the kernel lets us associate individual PIDs
>> > > with each cache. Since each vCPU is a PID, this means
>> > > we are able to allocate different cache size to
>> > > different CPUs. So we need to be able to represent
>> > > that in the XML. I think we should also represent
>> > > the allocation in a normal size (ie KiB), not in
>> > > count of min unit.
>> > >
>> > > So eg this shows allocating two cache banks and giving
>> > > one to the first 4 cpus, and one to the second 4 cpus
>> > >
>> > >   <cachetune>
>> > >      <bank type="l3" size="5632" unit="KiB" cpus="0,1,2,3"/>
>> > >      <bank type="l3" size="5632" unit="KiB" cpus="4,5,6,7"/>
>> > >   </cachetune>
>> > >
>> >
>> > I agree with your approach, we just need to keep in mind two more
>> > things.  I/O threads and the mail QEMU (emulator) thread can have
>> > allocations as well.  Also we need to say on which socket the allocation
>> > should be done.
>>
>> Also, I wonder if this is better put in the existing <cputune>
>> element, since this is really an aspect of the CPU configuration.
>>
>> Perhaps split configuration of cache banks from the mapping to
>> cpus/iothreads/emulator. Also, per Marcello's mail, we need to
>> include the host cache ID, so we know where to allocate from
>> if there's multiple caches of the same type. So XML could look
>> more like this:
>>
>>    <cputune>
>>        <cache id="1" host_id="2" type="l3" size="5632" unit="KiB"/>
>>        <cache id="2" host_id="4" type="l3" size="5632" unit="KiB"/>
>
>
>I don't think we require host_id here. we can only allow setting cache
>allocation only IF the VM has vcpu -> pcpu affinity setting. and let
>libvirt calculate where to set the cache (on which
>cache_id/resource_id/socket_id, the 3 ids are some meaning) since l3 caches
>are cpu's resource, only the VM running on specify cpu can benefit the
>cache.
>
>if we explicit allocate cache no care about what's the VM's pcpu affinity,
>helpless.
>

One thing we need to decide upfront is whether we are going to be fixing
user misconfiguration and to which extent because I feel like there's
too much discussion about that.  So either:

  a) We make sure that each thread that utilizes CAT is pinned to host
     threads without split cache, i.e. it cannot be scheduled outside of
     those.  I'm not using socket/core/thread and L3 because we need to
     be prepared here just in case any other cache hierarchy is used.

  b) We let user specify whatever they want.

Option (a) requires more code, more work, and must be checked on all
changes (vcpupin API, XML change, CPU hotplug, etc.), but option (b)
goes more with the rest of libvirt's config where we just let the users
shoot themselves in their feet by misconfiguration, i.e. if someone
wants to allocate cache on socket 0 and schedule all CPUs on socket 1,
then it's their fault.  Option (a) can save us some specification from
the XML, because we can compute some of the values.  However, that might
not be very reliable and we might end up requiring all the values
specified at the end anyway.  So from my point of view, I'd rather go
with (b) just so we don't swamp ourselves with the details, also we can
add the checks later.  And most importantly, as mentioned before, it
goes with the rest of the code.

>
>
>>
>>
>       <cpu_cache vcpus="0-3" id="1"/>
>>        <cpu_cache vcpus="4-7" id="2"/>
>>        <iothread_cache iothreads="0-1" id="1"/>
>>        <emulator_cache id="2"/>
>>    </cputune>
>>
>>
>> Regards,
>> Daniel
>> --
>> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/
>> :|
>> |: http://libvirt.org              -o-             http://virt-manager.org
>> :|
>> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/
>> :|
>>
>
>
>
>--
>Best regards
>- Eli
>
>天涯无处不重逢
>a leaf duckweed belongs to the sea , where not to meet in life
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: Digital signature
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20170112/085cba1e/attachment-0001.sig>