[Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework

[Jan Cholasta]

Hi,

On 13.6.2013 14:16, Petr Viktorin wrote:
> On 06/10/2013 04:48 PM, Petr Viktorin wrote:
>> On 05/31/2013 01:46 PM, Petr Viktorin wrote:
>>> Apply on top of my patches 0227-0234.
>>>
>>> These patches add an initial integration testing framework.
>>>
>>> Patch 0230 adds a plugin for ordered test classes.
>>> Nose orders methods within a test suite alphabetically, but we generally
>>> want to run them in the order defined. This adds the @ordered decorator
>>> that causes Nose to do just that, provided the plugin is loaded and
>>> enabled, and that the methods are defined in the same file. The
>>> ipa-run-tests wrapper is changed to enable the plugin.
>>> In the future we may want to use this for unit tests as well. It might
>>> also make sense to separate it from the FreeIPA project altogether.
>>>
>>> Patch 0231 adds configuration for tests. This reads environment
>>> variables like:
>>> - MASTER (FQDN of initial server)
>>> - REPLICA (space-separated FQDNs of replicas)
>>> - CLIENT (space-separated FQDNs of clients),
>>> - IPATEST_DIR (directory the tests use on the remote machines)
>>> etc., and loads them into an easy-to use Python object.
>>> A tool called ipa-test-config is provided that generates a full set of
>>> environment variables for shell-based tests from these, either global or
>>> specific for a given host.
>>> If environment variables don't work for us, alternate configuration
>>> methods can be added in the future.

I think you forgot to add "%dir 
%{python_sitelib}/ipatests/test_integration" to the spec file.

Is the "self = cls()" line at the beginning of Config.from_env() 
intentional?

+        self = cls()
+        env_normalize(env)
+
+        self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests',

>>>
>>> Patch 0232 adds an integration test framework.
>>> This extends Host object available from the configuration with methods
>>> to run commands and copy files on the remote host, and adds a base class
>>> for integration tests which can currently install and uninstall IPA in a
>>> "star" topology. (In the future, the install/uninstall code should also
>>> be made available as a shell command.)
>>> A simple test for user replication between two masters is provided.
>>> Log files from the remote hosts can be marked for collection, but the
>>> actual collection is left to a Nose plugin.
>>> The base class uses the @ordered decorator mentioned above.
>>>
>>> Patch 0233 improves on how commands are run on remote hosts.
>>> In the previous patch, the process's stdin and stdout were combined as a
>>> quick hack to avoid the problem that if we first read stdout and then
>>> stderr, then stderr's buffer can fill up and we'd deadlock (and the
>>> other way around). With this patch the streams are read in parallel.
>>> In the future this can be extended to calling whole commands in parallel
>>> (e.g. uninstalling IPA on all the hosts at once).
>>>
>>> Patch 0234 adds log collection to the BeakerLib integration plugin.
>>> This tars up the marked logs, downloads then, and calls rlFileSubmit on
>>> them.
>>>
>>> ---
>>
>> Attaching additional patches:
>>
>> Patch 0237 configures logging in ipa-run-tests to forward messages from
>> the IPA logging machinery to a normal Python logger. This way the logs
>> are captured
>> The logs are also printed to stderr so that there's some activity on the
>> terminal after you run the utility.
>>
>> Patch 0238 makes it possible to use RSA private SSH keys to log in to
>> the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and used if
>> $IPA_ROOT_SSH_PASSWORD is empty.
>> I've added this to the design page.

It seems that the code prefers password authentication over public key 
authentication if both are configured. IMO it would be better if it did 
the opposite.

>>
>> Patch 0239 makes test setup change the hostname, /etc/hosts, and
>> /etc/resolv.conf to match the configured values. These should be
>> equivalent to the fixes in
>> https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733
>>
>>
>> In test teardown, the changes are undone.
>
> I've rebased the patrchset and added small fixes for patches 0232 and 0239.
>
> New patch 0240 contains a few fixes/improvements to the Host class that
> were not trivial to rebase into previous patches.
>
> The WIP patch adds a sketch of some of the tests for CA-less
> (http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please
> comment if you can see where things can be improved for test authors!

Just a word of warning, there are still a few test cases I need to add 
to the CA-less test plan.

Honza

-- 
Jan Cholasta