[libvirt] [test-API] RFC: Stabilization of libvirt-test-API

[Guannan Ren]

On 04/04/2012 11:09 PM, Laine Stump wrote:
> On 03/29/2012 08:14 AM, Martin Kletzander wrote:
>> So here are the things I would like to do definitely (the optional
>> things follow later on):
>>   - fix hard-coded options into real options (e.g. commit 65449e)
> Forgive me if I'm suggesting things that already exist - I haven't had
> time to go through all of the test-API code, but want to make sure this
> gets brought up sooner rather than later.
>
> Two things that I think are very important for a general purpose test
> setup are:
>
> 1) The ability to have multiple physical machines, network devices of
> various types (bridges, standard NICs, sr-iov NICs) in the test harness
> available via standard names (implying, of course, that the
> address/capabilities (and even the existence/non-existence) of at least
> one other machine be configurable in a global config file referenced by
> every test).

              The env.cfg is the global config file that lists the 
default data needed by testcases.
              we could expand it for various requirements. The testsuit 
neither manage these devices
              or machines nor check if it is spelling-correct or present 
on testing machine right now.
              because it depends on testing environment , the tester 
should check it before writing it in
              env.cfg.


<snip>
              ##########################
              #
              # PCI device
              # a PIC device to use for attach/detach/reset tests
              # for example testpic = 00:19.0
              testpic =
</snip>


>
> 2) The ability to mark some tests as requiring certain standard objects
> from the global configuration (e.g. the second machine, a bridge
> interface, sr-iov NICs) and to either skip the test when some required
> object is missing, or fail the test (the test itself would be marked
> either OPTIONAL or REQUIRED).

            Currently, there is a file named BUGSKIP in the root of test-API
            If certain testcase is written in it for a bug related 
reason, the testcase
            will be skipped during running time, we could improve it for 
your purpose.
            There are 70~80 testcases. Any number of this testcases 
could be combined
             in different order according to various testing 
requirements. So there is not one
            shaped running model to go through all of testcases(we could 
have a basic one).
            there are some testcase file that are mostly used in "cases" 
folder in the root directory.



>
> This would allow us to have, for example, full testing of networking
> capabilities present in the test suite, but without raising an entry
> barrier for "random Joe" who only has a single machine, one NIC, etc.,
> but wants to run the test suite. When Joe ran the tests, each test that
> required multiple hosts (or sr-iov NICs or whatever esoteric piece of
> hardware/driver) and was designated as an "OPTIONAL" test, would be
> semi-silently skipped, but someone with a full complement of hardware
> who demanded a thorough and complete test could set a switch and
> guarantee that every single test would be run (or a failure saying, e.g.
> "object REMOTE_HOST required by this test is missing in config", or
> something like that).
          The current data list in env.cfg is default and satisfied for
        all of testcases.  "random Joe" have to know which testcases he 
want to
        run, if he need the testcase which rely on two NICs , he need to
        write MAC address of NICs to env.cfg before testing.



>
> Is there currently an allowance for these two items?
>
> --
> libvir-list mailing list
> libvir-list at redhat.com
> https://www.redhat.com/mailman/listinfo/libvir-list