[libvirt] [PATCH resend V10 03/12] Resctrl: Add new xml element to support cache tune

[Marcelo Tosatti]

On Wed, Mar 15, 2017 at 08:49:57PM +0100, Martin Kletzander wrote:
> On Wed, Mar 15, 2017 at 02:11:23PM -0300, Marcelo Tosatti wrote:
> >On Wed, Mar 15, 2017 at 03:59:31PM +0100, Martin Kletzander wrote:
> >>On Wed, Mar 15, 2017 at 02:23:26PM +0000, Daniel P. Berrange wrote:
> >>>On Wed, Mar 15, 2017 at 03:11:26PM +0100, Martin Kletzander wrote:
> >>>>On Mon, Mar 06, 2017 at 06:06:32PM +0800, Eli Qiao wrote:
> >>>>> This patch adds new xml element to support cache tune as:
> >>>>>
> >>>>> <cputune>
> >>>>>  ...
> >>>>>  <cachetune id='1' host_id='0' type='l3' size='2816' unit='KiB'
> >>>>
> >>>>Again, this was already discussed, probably, I just can't find the
> >>>>source of it.  But host_id actually already selects what cache is
> >>>>supposed to be used, so instead of type='l3' we only need scope='both'
> >>>>(or data/instruction) there.  Or am I missing something?  What I'm
> >>>>concerned about is that the host_id is mostly stable on one host (when
> >>>>the previously mentioned problems are fixed), but it will make no sense
> >>>>when the VM is migrated to another one.
> >
> >This is the same conditions as pinning a vcpu to a pcpu.
> >
> >So yes, it might be that you want to migrate to a host where
> >a different "host ID" number is used (which might or might not
> >be a different socket).
> >
> >>  Unfortunately, the only
> >>>>solution I can think of is using multiple keys to precisely describe the
> >>>>bank we want (e.g. host's cpu id, cache level and scope), but that seems
> >>>>very unclean.
> >>>
> >>>I tend to view use of this cachetune setting as being similar to
> >>>using host CPU passthrough - you're intentionally trading off
> >>>migratability of your  guest to get a perf boost.
> >>>
> >>>Even without the host_id bit, this is still non-portable, as you
> >>>might be requesting separate regions for code + data, but the
> >>>target host of migration may only support shared regions.
> >
> >Then migration should fail.
> >
> 
> Yes, it will automatically fail when you can't do the same allocations,
> that's easy.  The difference with host_id is that you cannot foresee
> whether keeping the host_id makes sense on the destination as well.

But on the destination, the user can use a different vcpu<->pcpu
configuration, right? 

That is, you do not enforce that if the source has say vcpu0 <-> pcpu2, 
the destination must have vcpu0 <-> pcpu2 (the configuration for that 
part is different).

So, the same thing can be applied to cache configuration.

> >>Sure, but those are things we can check during migration.  I'd be OK
> >>with disabling migration (or making it --unsafe) when migrating with
> >>cachetune.
> >
> >Migration support is required (for NFV usecase they want to migrate
> >VMs around).
> >
> >
> 
> So we at least need a check for the fact that either the caches are the
> somehow same or destination XML is supplied.
> 
> The other option would be what I hinted above, that is to change
> host_id= to something like level, scope, and pcpu(s)/socket(s).  

host_id maps to L3 socket via: 

commit d57e3ab7e34c51a8badeea1b500bfb738d0af66e
Author: Fenghua Yu <fenghua.yu at intel.com>
Date:   Sat Oct 22 06:19:50 2016 -0700

    x86/intel_cacheinfo: Enable cache id in cache info
    
    Cache id is retrieved from APIC ID and CPUID leaf 4 on x86.
    
    For more details please see the section on "Cache ID Extraction
    Parameters" in "Intel 64 Architecture Processor Topology
Enumeration".
    
    Also the documentation of the CPUID instruction in the "Intel 64 and
    IA-32 Architectures Software Developer's Manual"

So on the destination the user has to configure host_id= of the PCPU
which the guest is pinned to.

> That is
> fairly easy change from the code point of view, but I'm trying to think
> more about the user experience.
> 
> Martin