[libvirt] [Bug] make check fails when LIBVIRT_DEBUG is defined

[Daniel P. Berrange]

On Mon, Jun 27, 2016 at 01:23:46PM +0200, Michal Privoznik wrote:
> On 27.06.2016 10:05, Daniel P. Berrange wrote:
> > On Sat, Jun 25, 2016 at 04:32:21PM -0600, Eric Blake wrote:
> >> [moderator note: the mail server balked at 1.3 megabytes of attachments,
> >> so I've stripped them in my reply]
> >>
> >> On 12/31/1969 05:00 PM,  wrote:
> >>> Hello everybody,
> >>>
> >>> I set up a clean libvirt repo with:
> >>> git clone
> >>> ./autogen.sh
> >>> make -j5
> >>>
> >>> and then executed tests with various values of LIBVIRT_DEBUG using
> >>> the following command:
> >>> LIBVIRT_DEBUG=0…5 VIR_TEST_DEBUG=1 make check
> >>>
> >>> The logs are in this mail's attachments.
> >>>
> >>>
> >>> With LIBVIRT_DEBUG=0 (or =5 or probably any invalid value), the logs say:
> >>> "Ignoring invalid log level setting”
> >>> but still, 10 tests fail due to various reasons.
> >>>
> >>> With LIBVIRT_DEBUG=1, there is a slight difference. The number of tests
> >>> that fail is also 10, but here, commandtest fails instead of eventtest.
> >>>
> >>> With LIBVIRT_DEBUG=2, only 9 tests fail, because commandtest passes.
> >>>
> >>> With LIBVIRT_DEBUG=3, only 1 test fails: qemuargv2xmltest. The
> >>> reason is the same as in the previous tests.
> >>>
> >>> With LIBVIRT_DEBUG=4, it’s just qemuargv2xmltest that is failing,
> >>> but for a different reason than before:
> >>> "qemuParseCommandLineString should have logged a warning”.
> >>>
> >>>
> >>> Perhaps it’s okay that these tests fail, but still there's no harm
> >>> in letting people know.
> >>>
> >>> Thank you rbogorodskiy for guidance on the mailing list :)
> >>>
> >>> Have a nice day,
> >>> Tomasz
> >>>
> >>
> >> I don't know that anyone ever expected the testsuite to work with
> >> LIBVIRT_DEBUG set to anything (because turning on debug can change
> >> output of libvirt itself).  Things are different with VIR_TEST_DEBUG,
> > 
> > I'd certainly expect the test suite to pass 100% no matter what the
> > debug level is set to. Debug logging should not have a functional
> > effect on our code, besides outputting the log messages.
> 
> That's not 100% true. If we enable the debug messages, those are printed
> to stdout. Now we have some tests that expect certain output or some
> exact message to be logged. If we enable debug logs, the outputs are
> poisoned and don't match the expected values.
> I'd say this is a weakness of our test suite, not actual libvirt code.

Hmm, yes, unit tests really should not be expecting precise output on
stderr/stdout. Any that do are broken by design IMHO. At most they
should be doing a sub-string match if they want to look for something
and certainly not fail if extra output appears. Not least because I'll
often set LIBVIRT_DEBUG in order to debug the unit tests and would not
expect that to have side effects on them.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|