[Pulp-list] New Source Layout
Jay Dobies
jason.dobies at redhat.com
Wed Jun 13 13:03:13 UTC 2012
I think we're in a good place to get everyone back on master. The
migration process for your dev environment isn't the simplest, but
hopefully this is the last time we have to change the codebase to this
extent. I'll update the wiki for new developers later; this is meant for
those of us dealing with the transition.
Yes, this e-mail is long, even if you've come to expect that from me.
But please make sure you read this one. :)
= Migrate =
You're going to lose everything in this process, but ultimately it's a
dev environment, so you probably don't have much in there to begin with.
The first steps are going to be to remove the old Pulp dev install. From
what I hear, there's issues in pulp-dev's uninstall capabilities when
dealing with symlinks, so for simplicity we'll do a few manual steps:
1. $ sudo systemctl stop httpd.service
2. $ sudo rm -rf /etc/pulp/*
3. $ sudo rm -rf /var/lib/pulp/*
4. $ sudo rm /usr/bin/pulp-*
5. Drop your database entirely:
$ mongo
$ use pulp_database
$ db.dropDatabase()
Now for the new stuff. If you don't have any outstanding code, it might
be quicker to just move your code out to a temp location and re-clone
the repo. It will save you from having old stuff lying around.
1. $ git pull
The next few steps are to clear out things that git doesn't know about,
such as .pyc files. The following should be run from the git checkout
root. To be clear: do not do this if you have outstanding changes to
merge in. That should only be prad/john with plugin changes but everyone
think before you run these (all of this code has moved and you'll need
to merge your changes in elsewhere, more on that later).
2. $ rm -rf src # yes, you're deleting src in root; it's moved
3. $ rm -rf plugins
4. $ rm -rf extensions
Install the dev environment. You'll need to hit both setup.py files in
platform and rpm_support to get all of the site-packages references created.
5. $ sudo python ./pulp-dev.py -I
6. $ cd platform/src && sudo python ./setup.py develop
7. $ cd rpm-support/src && sudo python ./setup.py develop
At this point, things should largely be back to normal:
8. $ sudo pulp-migrate # note the version is back to 1 but I'll address
that soon
I found it easier to clean up my logs so I knew of any new issues coming up:
9a. (optional) $ rm -f /var/log/httpd/*
9b. (optional) $ rm -f /var/log/pulp/*
10. $ sudo systemctl start httpd.service
= Validation =
Keep an eye on /var/log/httpd/error_log to make sure the server starts
correctly. It won't if the plugins/type definitions aren't happy. If it
doesn't, ping me, but also check:
/var/lib/pulp/plugins/{distributors,importers,types}
to make sure there aren't any symlinks in there that don't point to
actual files.
= Running the Client =
Apache's mod_ssl seems to have changed a bit in Fedora 17. It used to
create a default cert with the CN "localhost.localdomain". Now, at least
on my box, the CN is the hostname of the box itself.
That's going to cause a problem when trying to run the client. The host
in admin.conf needs to match the CN. We ship with localhost.localdomain
by default but that's not sufficient anymore.
That means you'll need to change the host in /etc/pulp/admin/admin.conf
in order to do anything. The problem is that that file is symlinked into
the codebase, so if you edit it directly you may accidentally commit it.
If you create ~/.pulp/admin.conf it will use that as overrides
(~/.pulp/consumer.conf is the corresponding consumer override, but I may
remove that since it doesn't really make sense to scope anything
consumer related to a specific user). The contents should be:
[server]
host = sunflower
# always makes me smile that sayli named her desktop "sunflower" :)
= Where is Everything? =
I made a slight change to what was proposed last week. The bulk of our
codebase is organized into three subprojects:
platform - All of the plumbing, including server, client, and agent. If
you ever find yourself mentioning rpm/errata/kickstart/puppet/other_type
in here, it's wrong.
rpm-support - RPM pluigns/extensions/handlers. If you ever find yourself
mentioning rpm/errata/kickstart _outside_ of this directory, it's wrong.
builtins - The generic extensions/plugins/handlers that ship as part of
the Pulp platform. These cover things unrelated to types, such as the
create repo that doesn't imply an importer or simple login/logout
functionality.
In the future, we'll be adding subprojects for ISO support and puppet
support. Maybe even Windows support if Todd holds my children ransom and
forces me to.
Within each of the three subprojects there are a few consistent pieces:
src - Source code.
test/unit - Source code for unit tests
test/unit/data - Data needed for unit tests.
etc - Mimics the structure where any files are deployed.
For the plugin-related subprojects, you'll also find:
extensions
handlers
plugins/{distributors,importers,profilers,types}
= Tests =
What the above means is that we have test cases in three different
places now (one per subproject). This is actually a benefit in many
ways. During development in a particular subproject it's dirt simple and
quick as hell to run just the tests for that subproject. We're in a much
better position for test-driven development practices; we won't grow old
while running the full suite as a sanity check.
I'm serious about quicker. The platform tests run in 60-70 seconds. RPM
support runs in around 90. And after stripping out all of the untested
v1 code, we're floating around 71% total project code coverage. That's
actually pretty damn awesome. It's not gonna stop me from bitching that
we need to raise that number, but I do acknowledge that it's a very
healthy start.
Back to the three locations. There's a script in root called
run-tests.sh that runs the tests in all three locations and generates a
coverage report. The coverage report is under cover in the git root
(pull up index.html in there for the slick HTML view with the colored
markers of uncovered code).
That said, don't run that script yet. There's currently an issue in the
RPM support tests that cause them to barf a ton of directories into your
working directory and not clean them up, so running that script results
in a bunch of garbage thrown into your git checkout. We'll get those
fixed ASAP.
Something else to keep in mind. That script will run all three suites in
the same nosetests process. Jenkins, however, will intentionally run
them in three separate shells. My fear is that the single process run
will allow dependencies to bleed across subprojects and not be detected.
But at the same time, I dig the full project coverage report. So jenkins
will be our safety net for making sure the process divisions are in
place between tests in case you ever hit an issue where jenkins
indicates a failure but the run-tests.sh script does not.
More on testing in a future e-mail; I have a bunch of comments about
things we've done wrong in the past and how the new base classes work.
= Random Notes =
* pulp-consumer is broken right now. There are a few bugs in play. It's
always been using the admin certificate instead of its consumer
certificate, and when I fixed that it's now failing authentication. A
quick look at the code makes me think the auth code isn't looking at the
v2 consumers collection, but I really didn't look closely. I'll be
addressing this later today.
* I stripped out anything not currently being used. We still have it in
git history if we need to look again, but the goal here was to be as
clean as possible. That means things related to CDS, auditing, event
sending, the exporter, and the recent package migration script are all
deleted. We can always look up the v1 implementations when we go to
address them in v2 but I'd rather not have such a large sweeping clean
up again so took care of them now.
* Jeff's still working on the .spec files. There will be two: one for
Pulp platform and one for RPM support, each producing a number of RPMs.
I've seen the work done so far and they look great; much better
organization by created RPM rather than by spec section. Not sure of his
current status but I suspect we'll be building by the end of the week.
I think that's all I have for now. Ping me if you run into any problems
with the migration. I'm thinking of setting up an elluminate for the
whole day that I'll just sit in and people can jump in/screen share when
they run into specific problems, but we'll see how it goes.
--
Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org
More information about the Pulp-list
mailing list