[Pulp-list] Unit Test Notes

[Jay Dobies]

Out of everything done in the past two days on the git layout changes, 
the unit tests were in the worst shape.

To start with, the old testutil module has been renamed to base.py and 
moved into test/unit itself. That means no more boilerplate code to add 
../common to the source path before importing it. Just "import base" and 
move on.

Within base, there are now fewer TestCase subclasses to deal with.

Before I get into that, be sure to stop and think about your code 
organization and what it's doing. A lot of code does not require the 
database nor webservices and can be separated out. These tests don't 
need to subclass the server base class and incur the database setup 
penalties. This was the predominant cause for our tests taking in excess 
of 10 minutes prior to this cleanup; tests that simply checked a regex 
were hammering the database with initialization and clean up steps.

For example, the tests in rpm-support simply use unittest.TestCase since 
they don't need the database at all (feel free to create your own base 
class that configures logging if it makes sense).

There are currently three TestCase subclasses in base.py:

PulpServerTests - Used for anything that needs the database. This mainly 
means the managers if we're clean enough to separate out computational 
functions such as parsing and conversion methods. The setUp/tearDown 
cost isn't as much as it used to be, but it's still there.

PulpWebserviceTests - Used for testing controllers. This extends 
PulpServerTests and will get the database happy for you as well. The API 
is the same as it was with its self.{get,post,delete,etc} methods.

PulpClientTests - Ignores the database completely. In the setUp call it 
will populate a test client context for use in the test, mocking out the 
server and setting up a Recorder instance for the prompt to capture 
anything the prompt would have output to the screen and make it 
available for assertions.

Some other notes:

* When adding data to test/unit/data, please put everything related to a 
test in a directory with the same name. There were dozens of random 
directories and files in there, many of which were unused even before I 
started deleting stuff. It ends up being a mess to figure out which 
files belong to which test.

For an example of this, check out rpm-support/test/unit. There's a test 
case test_repo_cert_utils.py and a directory under data called 
/test_repo_cert_utils. No confusion.

* Clean up after yourself. Understand what files or database entries 
your tests are making and make sure to delete them.

* The above comment goes double for mocks. Trash them after a test if 
possible, but if you have to change something in Pulp as a whole be 
absolutely positive to undo your change afterwards. It's a nightmare 
debugging tests that pass individually but fail as a suite because one 
rogue test is changing a global state.


-- 
Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org