[libvirt] TSC frequency configuration & invtsc migration (was Re: [PATCH 4/4] kvm: Allow migration with invtsc)

[Eduardo Habkost]

On Thu, Jan 05, 2017 at 01:00:32PM +0000, Daniel P. Berrange wrote:
> On Thu, Jan 05, 2017 at 10:48:57AM -0200, Eduardo Habkost wrote:
> > On Thu, Jan 05, 2017 at 12:33:51PM +0000, Daniel P. Berrange wrote:
> > > On Thu, Jan 05, 2017 at 10:19:50AM -0200, Eduardo Habkost wrote:
> > > > On Thu, Jan 05, 2017 at 08:48:32AM -0200, Marcelo Tosatti wrote:
> > > > > On Wed, Jan 04, 2017 at 11:36:31PM -0200, Eduardo Habkost wrote:
> > [...]
> > > > > > Scenario 2:
> > > > > > 
> > > > > > Host A has TSC scaling, host B doesn't have TSC scaling. We want
> > > > > > to be able to start the VM on host A, and migrate to B. In this
> > > > > > case, the only possible solution is to use B's frequency when
> > > > > > starting the VM. The QEMU process doesn't have enough information
> > > > > > to make that decision.
> > > > > 
> > > > > That is a good point. But again, its a special case and 
> > > > > should be supported by -cpu xxx,tsc-frequency=zzzz.
> > > > > 
> > > > > However, for the vast majority of 99.999% cases, the issue
> > > > > can be handled entirely in QEMU, without libvirt involvement,
> > > > > and without adding extra steps to the management software.
> > > > 
> > > > I agree it should cover most cases. The only problem here is that
> > > > it can break migration in unexpected ways.
> > > > 
> > > > Then my point is: assuming that libvirt will prefer to require
> > > > explicit TSC frequency configuration to enable invtsc migration
> > > > (instead of getting unpredictable migration compatibility), is
> > > > the added complexity to migration code worth the effort, if
> > > > choosing an explicit frequency is safer and more predictable? I
> > > > believe this is where we disagree.
> > > 
> > > I believe that if libvirt detects that QEMU supports the 'tsc-frequency'
> > > option, then libvirt should set it by default in the XML, if not already
> > > set by the mgmt app. That way, libvirt can validate TSC freq comapt
> > > for migration before it even launches QEMU in the target host.
> > 
> > If you do this unconditionally, you have another problem: if
> > tsc-frequency is set explicitly, migration is only possible if
> > TSC frequency of the destination matches[1], or if TSC scaling is
> > supported by the destination. It's a good idea to set a TSC
> > frequency only if invtsc is enabled explicitly in the config.
> 
> If we don't set tsc-frequency and the TSC frequency doesn't
> match, does that mean the guest migration succeed, but suddenly
> sees different TSC frequency ?

If TSC scaling is unavailable, yes. If the destination host
supports TSC scaling, we automatically keep the original TSC
frequency on migration.

> 
> I guess we we allowed that historically we can't break that
> now, so setting it only if invtsc is set seems reasonable.

I don't think we really had a choice, KVM would be much less
useful if we didn't allow migration between hosts with different
frequencies.

> 
> > 
> > [1] Currently the frequency needs to match exactly. That's a
> > separate issue: we should probably add a knob to allow a slight
> > variation in TSC frequency (e.g. <1% difference).
> 
> Regards,
> Daniel
> -- 
> |: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org              -o-             http://virt-manager.org :|
> |: http://entangle-photo.org       -o-    http://search.cpan.org/~danberr/ :|

-- 
Eduardo