[Freeipa-devel] [RFE] Integration testing

[Jakub Hrozek]

On Mon, Jun 03, 2013 at 02:05:09PM +0200, Petr Viktorin wrote:
> Hello,
> A design document for integration testing is available at
> http://www.freeipa.org/page/V3/Integration_testing. I've copied it below for
> easier quoting.
> 
> 
> __NOTOC__
> 
> = Overview =
> 
> Make it possible to write and run multi-host integration tests (such as:
> install master & replica, add user on replica, verify it's added on master).
> 
> These tests will be run from continuous integration.
> Any developer can also run them manually.
> 
> = Use Cases =
> 
> == Continuous integration ==
> 
> The developer team at Red Hat will run a Jenkins continuous integration
> server
> that will run the tests automatically (after each commit if resources are
> available).
> 
> The CI results will be posted publicly.
> 
> == Developer testing ==
> 
> Anyone is be able to run integration tests without advanced infrastructure,
> only a number of virtual machines to run the tests on is needed.
> 
> == Beaker integration ==
> 
> The tests will run seamlessly inside [http://beaker-project.org/
> Beaker]/[https://fedoraproject.org/wiki/QA/RHTS RHTS].
> A special option enables reporting via BeakerLib.
> 
> = Non-goals =
> 
> A complete testing/continuous integration setup needs some steps that will
> not
> be included in IPA's test suite:
> 
> * Building the code
> * VM provisioning
>     <!--
>     There are just too many disparate ways to do it; people
>     with a virtual datacenter should already have a preferred tool.
>     If we come up with something for ourselves we'll have to make too many
>     assumptions for it to be useful somewhere else.
>     -->
> * Configuring the basic system, installing the packages
>     <!--
>     Again support for this can be added in the future.
>     (Release Puppet/Ansible configuration?)
>     -->
> 
> = Design=
> 
> The Python package with the IPA test suite is renamed to <tt>ipatests</tt>,
> and
> packaged for RPM-based systems as <tt>freeipa-tests</tt>.
> Eventually the package will be included in Fedora.
> 
> Integration tests will be controlled from a single machine, and executed
> on a number of "remote" machines that act as servers, replicas, clients,
> etc.
> The controlling machine communicates with the others via the SSH protocol.
> (The controlling machine may be the same as one of the "remote" ones.)
> 
> Integration tests are included in the main IPA set suite, and configured
> using
> environment variables. If the variables are missing, all integration tests
> are
> skipped.
> If an insufficient number of hosts is configured for a test, the individiual
> test will be skipped.
> 
> A tool is provided to run installed tests.
> 
> The remote machines used for integration testing are required to have
> relevant
> IPA packages installed, firewall opened up, any needed workarounds applied
> (RPM
> downgrades, SELinux mode,...), and sshd set up to allow root login.
> The test runner will connect to these machines, install IPA, perform the
> tests,
> and then uninstall IPA & return the systems to their previous state.
> 
> A plugin for integration with BeakerLib is provided.
> 
> = Test configuration =
> 
> Tests are configured using these environment variables.
> 
> == Host configuration ==
> 
> ; $MASTER
> :  FQDN of the first IPA server
> ; $REPLICA
> :  FQDNs of other IPA servers (space-separated)
> ; $CLIENT
> :  FQDNs of IPA clients (space-separated)
> ; $MASTER_env2, $REPLICA_env2, $CLIENT_env2, $MASTER_env3, ...
> :  can be used for additional domains when needed
> 
> DNS needs to be set up so that IP addresses can be obtained for these hosts.
> 
> == Basic configuration ==
> 
> ; $IPATEST_DIR
> :  Directory for test data on the remote hosts
> :  Default: /root/ipatests
> ; $DNSFORWARD
> :  IP of a DNS forwarder
> :  Default: 8.8.8.8
> 
> == Test customization ==
> 
> ; $DOMAIN
> :  IPA domain name
> :  Default: taken from $MASTER
> ; $NISDOMAIN
> :  NIS domain name
> :  Default: ipatest
> ; $NTPSERVER
> :  NIS domain name
> :  Default: ipatest
> ; $IPv6SETUP
> :  Set to TRUE for IPv6-only connectivity
> ; $IPADEBUG
> :  Set to enable test debugging
> 
> ; $ADMINID
> :  Admin username
> :  Default: admin
> ; $ADMINPW
> :  Admin user password
> :  Default: Secret123
> ; $ROOTDN
> :  Directory manager DN
> :  Default: cn=Directory Manager
> ; $ROOTDNPWD
> :  Directory manager password
> :  Default: Secret123
> 
> = Supporting tools =
> 
> == ipa-test-config ==
> 
> This tool reads the configuration variables above and outputs a Bash script
> that sets a much more complete set of variables for easy shell-based testing
> or test set-up.
> 
> Without arguments, <tt>ipa-test-config</tt> outputs information specific
> to the host it is run on. When given a hostname, it prints config for that
> host.
> With the <tt>--global</tt> flag, it outputs configuration common to all
> hosts.
> 
> == ipa-run-tests ==
> 
> This tool is a wrapper arount <tt>nosetests</tt> and accepts the same
> arguments
> as Nose.
> It loads any additional plugins and runs tests from the system-installed IPA
> test suite.
> 
> == Other ==
> 
> TBD: Additional command-line tools may be provided for tasks such as
> installing
> IPA in a given topology.
> 
> = Implementation =
> 
> Test cases are implemented as Nose test classes, with
> installation/uninstallation as class setup/teardown.
> 
> A BeakerLib plugin is provided that starts/ends Beaker phases for Nose test
> contexts and cases, issues a Beaker assertion (rlPass/rlFail) for each test
> case, and collects and submits relevant logs.
> 
> A separate plugin will be provided to collect logs outside of a Beaker
> environment.
> 
> = Example instructions =
> To run the test called <tt>test_integration/test_simple_replication.py</tt>,
> which needs to run with two masters, follow these instructions.
> 
> Install the IPA server packages on two machines, and do any preparations
> necessary to install IPA (e.g. configure/disable the firewall).
> 
> Then, install the <tt>freeipa-tests</tt> package on the machine that will
> run
> the tests (this may be one of the machines above, or preferably a different
> one).
> Set MASTER and REPLICA environment variables to the fully qualified
> hostnames
> of the two machines prepared earlier.
> Also set any other relevant variables listed in
> [[#Test configuration|Test configuration]].
> You may run <tt>ipa-test-config --global</tt> to verify how the test
> configuration will be handled.
> 
> The next steps depend on whether the test will run within a BeakerLib
> session
> or not.

How much is the test library tied to IPA? I'd like to reuse it for SSSD
tests, obviously those tests that would be able to run against an IPA
server could be easy, but what about tests that require some other
custom LDAP server? Would the library allow rolling out a more generic
server or would we need to set up a "stable" machine which the tests
would connect to?