[Freeipa-devel] [PATCHES] 0230-0244 Integration testing framework

[Petr Viktorin]

On 07/11/2013 01:28 PM, Jan Cholasta wrote:
> On 10.7.2013 17:50, Petr Viktorin wrote:
>> On 07/10/2013 02:03 PM, Jan Cholasta wrote:
>>> make test seems to run fine with patches 230-242 applied, however
>>> ipa-run-tests produces the following output:
>>>
>> [...Skipping nose output...]
>>>
>>>
>>> I guess the location of the test certificate should be made configurable
>>> in order to fix the host and service plugin test failures.
>>
>> Making the existing test suite pass out of tree it's not a goal for this
>> patchset. There is a pending patch for the service cert.
>
> Better report it now than forget it later.
>
>>
>>> Also, there is a lot of debugging messages in ipa-run-tests output which
>>> wasn't there before IIRC, is that intentional?
>>
>> Yes, I believe that for the integration tests it's better to see what is
>> going on. Both for manual runs, and also this way the information is
>> more easily picked up by CI tools.
>
> It's harder to see what's going on with so much noise IMHO, I would
> prefer less verbose output for manual runs by default.

On the other hand, the integration tests take too long and without 
output they may seem stuck. I'd rather not make that the default.
I'll try to find some balance and make it more easily configurable in 
future patches.

>>> If $MASTER (and possible other host names) is not resolvable,
>>> ipa-test-config --global crashes:
>>>
>>> $ ipa-test-config --global
>>> Traceback (most recent call last):
>>>    File "/usr/bin/ipa-test-config", line 104, in <module>
>>>      print main(sys.argv[1:]),
>>>    File "/usr/bin/ipa-test-config", line 58, in main
>>>      return config.env_to_script(get_object(conf,
>>> args).to_env(**kwargs))
>>>    File
>>> "/usr/lib/python2.7/site-packages/ipatests/test_integration/config.py",
>>> line 168, in to_env
>>>      env['MASTER'] = default_domain.master.hostname
>>>    File
>>> "/usr/lib/python2.7/site-packages/ipatests/test_integration/config.py",
>>> line 282, in master
>>>      return self.masters[0]
>>> IndexError: list index out of range
>>
>> Fixed.
>> Now, if the name is not resolvable, reading the config will fail.
>> Alternatively, the IP address can be given in variables such as
>> $BEAKERREPLICA1_IP_env1 (no, I didn't invent the name).
>
> This fixed ipa-test-config as advertised, but "ipa-run-tests
> test_integration/test_simple_replication.py" still fails.
>
>>> Both the tests in test_simple_replication.py fail for me. I suspect it
>>> is because the data isn't replicated fast enough, a little delay between
>>> user-show and user-add might fix this.
>>
>> Hm, they worked for me.
>>   I've added a delay. It seems fragile, I wonder what the proper way to
>> do this would be.
>
> Hmm, the "replica -> master" test still fails for me, even with delay
> increased to 20 s.

Is there anything in the DS logs?

> Something less fragile than delay would certainly be nice.

I'll read up on the DS replication docs, and ask for some guidance.

>> I've made some other changes, mainly BeakerLib plugin output.
>>
>> Patch 240: Avoid infinite recursion that happened with some cases of bad
>> SSH credentials
>> Patch 241-243: Rework the BeakerLib plugin output to better match
>> traditional Beaker tests
>> Patch 244: Make it possible to explicitly specify IP addresses of hosts
>
> To sum things up, there are still some little bugs, but these can be
> fixed after the beta release, in general everything seems to work, so ACK.

Thanks, pushed to master: 23d3fde05960d6e47485e246e11bec701b30cb59
Not closing the ticket yet until the issues are cleared (or moved to 
separate tickets).

-- 
Petr³