[Freeipa-devel] [PATCHES] 0227-0229 freeipa-tests package & Beaker integration plugin

Petr Viktorin pviktori at redhat.com
Mon Jun 17 15:08:56 UTC 2013 

On 06/17/2013 04:44 PM, Jan Cholasta wrote:
> On 17.6.2013 15:09, Jan Cholasta wrote:
>> On 17.6.2013 14:39, Jan Cholasta wrote:
>>> On 14.6.2013 16:01, Petr Viktorin wrote:
>>>> On 06/14/2013 03:20 PM, Jan Cholasta wrote:
>>>>> Hi,
>>>>>
>>>>> On 28.5.2013 17:55, Petr Viktorin wrote:
>>>>>> Hello,
>>>>>>
>>>>>> Patch 0227 creates the freeipa-tests package.
>>>>>> As a system package, it needs a more unique name than "tests", so I
>>>>>> renamed it to "ipatests". I also changed imports and references to
>>>>>> it.
>>>>>> Sorry to everyone developing tests right now ­– there will be
>>>>>> conflicts,
>>>>>> but hopefully they'll be straightforward.
>>>>>> Note that the test suite does not yet pass when run outside the Git
>>>>>> tree. Work on that is ongoing but not a priority right now (it's more
>>>>>> important to get some integration tests running). Help would be
>>>>>> appreciated :)
>>>>>
>>>>> Typo in commit message: "Tename the 'tests' directory ..."
>>>>
>>>> Thanks for the catch!
>>>>
>>>>> The patch needs rebasing.
>>>>
>>>> Attaching rebased versions.
>>>>
>>>>>> Patch 0228 adds a wrapper based on make-test which runs the
>>>>>> system-installed test suite. freeipa-tests installs it as
>>>>>> /usr/bin/ipa-run-tests.
>>>>>> As I said above the tests currently fail when run this way.
>>>>>>
>>>>>> Patch 0229 adds a Nose plugin for integration with BeakerLib[1]. When
>>>>>> the plugin is loaded (ipa-run-tests does that) and enabled (using the
>>>>>> --with-beakerlib option), it hooks into Nose and runs rlPhase*,
>>>>>> rlPass,
>>>>>> rlFail and rlLog* Bash functions at appropriate events.
>>>>>>
>>>>>
>>>>> I still need to actually run the code, I will do that with your
>>>>> patches
>>>>> 230-240 included.
>>>>
>>>> If I may suggest, please test this set by itself. The tests→ipatests
>>>> rename blocks or conflicts with other patches so it would be good to
>>>> get
>>>> it in soon, without waiting on the integration testing review.
>>>>
>>>>
>>>
>>> When I run "make test" I get the following error:
>>>
>>> Traceback (most recent call last):
>>>    File "/usr/lib/python2.7/site-packages/nose/loader.py", line 413, in
>>> loadTestsFromName
>>>      addr.filename, addr.module)
>>>    File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in
>>> importFromPath
>>>      return self.importFromDir(dir_path, fqname)
>>>    File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in
>>> importFromDir
>>>      mod = load_module(part_fqname, fh, filename, desc)
>>>    File
>>> "/home/jcholast/freeipa/ipatests/test_xmlrpc/test_host_plugin.py", line
>>> 59, in <module>
>>>      fd = open('tests/test_xmlrpc/service.crt', 'r')
>>> IOError: [Errno 2] No such file or directory:
>>> 'tests/test_xmlrpc/service.crt'
>>>
>>> I believe the certificate path should begin with "ipatests" instead of
>>> "tests".

Fixed, thanks.

>>> When I run "ipa-run-tests", I get an additional similar error in
>>> test_service_plugin.py:
>>>
>>> Traceback (most recent call last):
>>>    File "/usr/lib/python2.7/site-packages/nose/loader.py", line 413, in
>>> loadTestsFromName
>>>      addr.filename, addr.module)
>>>    File "/usr/lib/python2.7/site-packages/nose/importer.py", line 47, in
>>> importFromPath
>>>      return self.importFromDir(dir_path, fqname)
>>>    File "/usr/lib/python2.7/site-packages/nose/importer.py", line 94, in
>>> importFromDir
>>>      mod = load_module(part_fqname, fh, filename, desc)
>>>    File
>>> "/usr/lib/python2.7/site-packages/ipatests/test_xmlrpc/test_service_plugin.py",
>>>
>>>
>>> line 42, in <module>
>>>      fd = open('ipatests/test_xmlrpc/service.crt', 'r')
>>> IOError: [Errno 2] No such file or directory:
>>> 'ipatests/test_xmlrpc/service.crt'
>>>
>>>
>>> Also with "ipa-run-tests", many of the cmdline tests are failing with:
>>>
>>> Traceback (most recent call last):
>>>    File "/usr/lib/python2.7/site-packages/nose/case.py", line 381, in
>>> setUp
>>>      try_run(self.inst, ('setup', 'setUp'))
>>>    File "/usr/lib/python2.7/site-packages/nose/util.py", line 469, in
>>> try_run
>>>      return func()
>>>    File
>>> "/usr/lib/python2.7/site-packages/ipatests/test_cmdline/cmdline.py",
>>> line 58, in setUp
>>>      'Command %r not available' % self.command
>>> AssertionError: Command 'ipa-client/ipa-getkeytab' not available

Yes, some tests don't work out-of-tree yet. This will be addressed in 
future patches.
The important thing to check is if the beakerlib plugin handles failures 
well.

>> ipa-run-tests --with-beakerlib is horribly slow for me, is that expected?

Yes. For every logged line, BeakerLib's default logging backend starts 
up Python, parses a XML file, appends the line, and writes the XML out 
again. So especially with longer runs it's really slow.

> Also I get the following failures with --with-beaker (in addition to the
> previously reported ones):
>
> ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>
> :: [   LOG    ] :: Nose test: test_help.test_ipa_help
> ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>
>
> :: [  ERROR   ] :: [ipa.ipalib.cli.cli] non-public: AttributeError:
> 'API' object has no attribute 'parser'
> :: [  ERROR   ] :: Traceback (most recent call last):
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 129, in execute
> :: [  ERROR   ] ::     result = self.Command[_name](*args, **options)
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipalib/frontend.py", line 435, in
> __call__
> :: [  ERROR   ] ::     ret = self.run(*args, **options)
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 765, in run
> :: [  ERROR   ] ::     self.api.parser.print_help(outfile)
> :: [  ERROR   ] :: AttributeError: 'API' object has no attribute 'parser'
> :: [  ERROR   ] :: Traceback (most recent call last):
> :: [  ERROR   ] ::   File "/usr/lib64/python2.7/unittest/case.py", line
> 369, in run
> :: [  ERROR   ] ::     testMethod()
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
> :: [  ERROR   ] ::     self.test(*self.arg)
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py",
> line 63, in test_ipa_help
> :: [  ERROR   ] ::     return_value = api.Backend.cli.run(['help'])
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipalib/cli.py", line 1067, in run
> :: [  ERROR   ] ::     result = self.execute(name, **kw)
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipalib/backend.py", line 146, in execute
> :: [  ERROR   ] ::     raise error #pylint: disable=E0702
> :: [  ERROR   ] :: InternalError: an internal error has occurred
> :: [   FAIL   ] :: Test failed: unhandled exception
> :: [   LOG    ] :: Duration: 1s
> :: [   LOG    ] :: Assertions: 0 good, 1 bad
> :: [   FAIL   ] :: RESULT: Nose test: test_help.test_ipa_help
>
> ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>
> :: [   LOG    ] :: Nose test: test_help.test_ipa_without_arguments
> ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
>
>
> :: [  ERROR   ] :: Traceback (most recent call last):
> :: [  ERROR   ] ::   File "/usr/lib64/python2.7/unittest/case.py", line
> 369, in run
> :: [  ERROR   ] ::     testMethod()
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
> :: [  ERROR   ] ::     self.test(*self.arg)
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py",
> line 71, in test_ipa_without_arguments
> :: [  ERROR   ] ::     api.Backend.cli.run([])
> :: [  ERROR   ] ::   File
> "/usr/lib/python2.7/site-packages/ipatests/test_cmdline/test_help.py",
> line 55, in __exit__
> :: [  ERROR   ] ::     assert isinstance(exc_value, self.exception),
> exc_value
> :: [  ERROR   ] :: AssertionError: 'API' object has no attribute 'parser'
> :: [   FAIL   ] :: Test failed
> :: [   LOG    ] :: Duration: 0s
> :: [   LOG    ] :: Assertions: 0 good, 1 bad
> :: [   FAIL   ] :: RESULT: Nose test: test_help.test_ipa_without_arguments

We can fix individual out-of-tree failures later, the priority now is 
that in-tree tests are not broken, and that the beakerlib plugin works.

-- 
Petr³

-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0227.3-Make-an-ipa-tests-package.patch
Type: text/x-patch
Size: 71543 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130617/eed04183/attachment.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0228.3-Add-ipa-run-tests-command.patch
Type: text/x-patch
Size: 2517 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130617/eed04183/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: freeipa-pviktori-0229.3-Add-Nose-plugin-for-BeakerLib-integration.patch
Type: text/x-patch
Size: 11056 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/freeipa-devel/attachments/20130617/eed04183/attachment-0002.bin>

More information about the Freeipa-devel mailing list