[libvirt] [PATCH 2/6] tests: qemuxm2argv: Add infrastructure for testing with real qemuCaps

[Peter Krempa]

On Sun, Apr 15, 2018 at 15:29:23 +0200, Ján Tomko wrote:
> On Wed, Apr 11, 2018 at 11:16:27AM -0400, John Ferlan wrote:
> > 
> > 
> > On 04/04/2018 04:13 AM, Peter Krempa wrote:
> > > Allow testing of XML->argv conversion with using a real capability map
> > > as used in the qemucapabilitiestest. This allows specifying the required
> > > qemu version with the test rather than having to enumerate all the
> > > required capabilities.
> > > 
> > > Signed-off-by: Peter Krempa <pkrempa at redhat.com>
> > > ---
> > >  tests/qemuxml2argvtest.c | 31 +++++++++++++++++++++++++++++++
> > >  1 file changed, 31 insertions(+)



> > Perhaps the macro(s) should be DO_TEST_VERS[ION] or DO_TEST_<ARCH>_VERS
> > instead since DO_TEST runs the test with specific capabilities and this
> > new model runs the test with a specific version for a specific platform.
> > 
> > How do or should we enforce that when adding a new test from this point
> > forward that the DO_TEST_CAPS model is chosen over DO_TEST?  I suppose
> > there is still value in DO_TEST since one can pick and choose which CAP
> > is needed - although I'll freely admit that's usually a cut-n-paste from
> > some other test and change the name type activity.
> 
> The problem with that approach is that we're testing against
> capability combinations that might never occur in the wild.
> 
> I actually remember fixing some bug that was supposed to be covered by
> a unit test, but it was missing the important capability.

Actually a very good example is the recent series of adding the
'discard-data' attribute to memory devices. Since the new feature is
decided with capability bits, all existing tests were still passing, but
the attribute would be added to NVDIMM memory devices which would delete
the contents.

I think we also need to add a way to take the latest caps present in the
repo to be used with the bulk of tests rather than binding it to a
specific version.

That way we can see whether the command line is stable when we add a
capability based test.

If the command line will change, the test case then can be split for the
specific version of qemu capabilities, so that the contemporary test
case still captures the new case, but the old behaviour is kept too.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/libvir-list/attachments/20180417/d5a87afa/attachment-0001.sig>