[Freeipa-devel] [PATCHES] 0230-0240 Integration testing framework

[Jan Cholasta]

On 14.6.2013 16:19, Jan Cholasta wrote:
> Hi,
>
> On 13.6.2013 14:16, Petr Viktorin wrote:
>> On 06/10/2013 04:48 PM, Petr Viktorin wrote:
>>> On 05/31/2013 01:46 PM, Petr Viktorin wrote:
>>>> Apply on top of my patches 0227-0234.
>>>>
>>>> These patches add an initial integration testing framework.
>>>>
>>>> Patch 0230 adds a plugin for ordered test classes.
>>>> Nose orders methods within a test suite alphabetically, but we
>>>> generally
>>>> want to run them in the order defined. This adds the @ordered decorator
>>>> that causes Nose to do just that, provided the plugin is loaded and
>>>> enabled, and that the methods are defined in the same file. The
>>>> ipa-run-tests wrapper is changed to enable the plugin.
>>>> In the future we may want to use this for unit tests as well. It might
>>>> also make sense to separate it from the FreeIPA project altogether.
>>>>
>>>> Patch 0231 adds configuration for tests. This reads environment
>>>> variables like:
>>>> - MASTER (FQDN of initial server)
>>>> - REPLICA (space-separated FQDNs of replicas)
>>>> - CLIENT (space-separated FQDNs of clients),
>>>> - IPATEST_DIR (directory the tests use on the remote machines)
>>>> etc., and loads them into an easy-to use Python object.
>>>> A tool called ipa-test-config is provided that generates a full set of
>>>> environment variables for shell-based tests from these, either
>>>> global or
>>>> specific for a given host.
>>>> If environment variables don't work for us, alternate configuration
>>>> methods can be added in the future.
>
> I think you forgot to add "%dir
> %{python_sitelib}/ipatests/test_integration" to the spec file.
>
> Is the "self = cls()" line at the beginning of Config.from_env()
> intentional?
>
> +        self = cls()
> +        env_normalize(env)
> +
> +        self = cls(test_dir=env.get('IPATEST_DIR') or '/root/ipatests',
>

Also typo in commit message: "Integration tests are be configured ..."

>>>>
>>>> Patch 0232 adds an integration test framework.
>>>> This extends Host object available from the configuration with methods
>>>> to run commands and copy files on the remote host, and adds a base
>>>> class
>>>> for integration tests which can currently install and uninstall IPA
>>>> in a
>>>> "star" topology. (In the future, the install/uninstall code should also
>>>> be made available as a shell command.)
>>>> A simple test for user replication between two masters is provided.
>>>> Log files from the remote hosts can be marked for collection, but the
>>>> actual collection is left to a Nose plugin.
>>>> The base class uses the @ordered decorator mentioned above.
>>>>
>>>> Patch 0233 improves on how commands are run on remote hosts.
>>>> In the previous patch, the process's stdin and stdout were combined
>>>> as a
>>>> quick hack to avoid the problem that if we first read stdout and then
>>>> stderr, then stderr's buffer can fill up and we'd deadlock (and the
>>>> other way around). With this patch the streams are read in parallel.
>>>> In the future this can be extended to calling whole commands in
>>>> parallel
>>>> (e.g. uninstalling IPA on all the hosts at once).
>>>>
>>>> Patch 0234 adds log collection to the BeakerLib integration plugin.
>>>> This tars up the marked logs, downloads then, and calls rlFileSubmit on
>>>> them.

Missing space in commit message: "... log files fromthe remote ..."

>>>>
>>>> ---
>>>
>>> Attaching additional patches:
>>>
>>> Patch 0237 configures logging in ipa-run-tests to forward messages from
>>> the IPA logging machinery to a normal Python logger. This way the logs
>>> are captured
>>> The logs are also printed to stderr so that there's some activity on the
>>> terminal after you run the utility.
>>>
>>> Patch 0238 makes it possible to use RSA private SSH keys to log in to
>>> the remote machines. The key is given in $IPA_ROOT_SSH_KEY, and used if
>>> $IPA_ROOT_SSH_PASSWORD is empty.
>>> I've added this to the design page.
>
> It seems that the code prefers password authentication over public key
> authentication if both are configured. IMO it would be better if it did
> the opposite.
>
>>>
>>> Patch 0239 makes test setup change the hostname, /etc/hosts, and
>>> /etc/resolv.conf to match the configured values. These should be
>>> equivalent to the fixes in
>>> https://github.com/freeipa/tests/blob/master/ipa-tests/beaker/ipa-server/shared/ipa-install.sh#L733
>>>
>>>
>>>
>>> In test teardown, the changes are undone.
>>
>> I've rebased the patrchset and added small fixes for patches 0232 and
>> 0239.
>>
>> New patch 0240 contains a few fixes/improvements to the Host class that
>> were not trivial to rebase into previous patches.
>>
>> The WIP patch adds a sketch of some of the tests for CA-less
>> (http://www.freeipa.org/page/V3/CA-less_install#Test_Plan). Please
>> comment if you can see where things can be improved for test authors!
>
> Just a word of warning, there are still a few test cases I need to add
> to the CA-less test plan.
>
> Honza
>


-- 
Jan Cholasta