[libvirt] Scope of the test driver? (Was: Re: [PATCH] test_driver: implement virDomainSendKey)

[Daniel P. Berrangé]

On Mon, Jun 03, 2019 at 12:31:56PM +0200, Peter Krempa wrote:
> On Mon, Jun 03, 2019 at 08:52:55 +0200, Erik Skultety wrote:
> > On Mon, Jun 03, 2019 at 08:40:13AM +0200, Pavel Hrdina wrote:
> > > On Mon, Jun 03, 2019 at 08:23:57AM +0200, Erik Skultety wrote:
> > > > On Sat, Jun 01, 2019 at 02:46:56PM +0200, Ilias Stamatis wrote:
> 
> [...]
> 
> > > > I think this API should be merely about translating the value, because
> > > > this way you're just blocking a synchronous API for no apparent benefit. I
> > > > believe it should return instantaneously. On the other hand, I'm just wondering
> > > > what value it brings having an API translating keycodes, but I guess for
> > > > consistency reasons, we can happily introduce it.
> > >
> > > The benefit is to simulate the holdtime as for example hyperv driver is
> > > doing, in QEMU we just pass everything to the QEMU itself where that one
> > > will probably do similar operation.
> > 
> > Both hyperv and vbox have to do it because they don't support it, so they do
> > usleep before sending down and up keycodes respectively, QEMU does it
> > iternally. In test driver, you're not testing any true codes, so doing plain
> > usleep si IMHO wrong, from app dev's perspective you're only testing whether
> > usleep works by providing arbitrary time to wait, which raises the obvious
> > question "why".
> 
> I think the problem here is that it's unclear what the purpose of the
> test driver is supposed to be. Clarifying the purpose first might have
> positive impact on the design decisions here.
> 
> So what's the point of the test driver?
> 
> Is it meant for human interaction, e.g. for application developers or
> users wanting to test stuff? In that case you probably want to simulate
> the "worst" behaviour we have, which would be the blocking timeout so
> that users/devs can see the worst case implications of running such a
> command.
> 
> Alternatively the purpose may be to allow some kind of "unit" testing
> for APPs using libvirt. (I specifically used "unit" testing, because any
> kind of sane real integration testing will not use the test driver
> because it would not accomplish much). In such case the timeout is a
> hassle because it just unnecessarily prolongs test runs and does not add
> much value.
> 
> Any other ideas? I don't have any more.

The "test" driver essentially exist to allow application developers to
do automated functional testing of their applications. ie it should allow
them to exercise any libvirt API calls they make and get some kind of
interesting results back, without needing full VM actually running.

It is not expected to perfectly replicate what you'd see when using the
QEMU driver with a real VM in terms of performance / scalability, nor
expose every possible error condition / behaviour. If that's what you are
expecting, then you need full stack integration testing, not functional
testing.

Essentially think of the test driver as allowing app developers to do a
basic evaluation to validate their app isn't stupidly broken in some
way.

The reason why you'd use the test driver for functional tests instead
doing a full integration test is that it has no further external
dependancies, either software or hardware. This makes it something you
can always run at any time, any place.

Full integration testing is better but you'll need a much more defined
stack of softrware & hardware that can't be assumed to be always present.

With this in mind I do *not* think it is needed to actually have a sleep()
to match the "hold time". If you want that kind of accuracy of testing,
do integration tests with a real VM running.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|