[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

[Pulp-list] New Source Layout



I think we're in a good place to get everyone back on master. The migration process for your dev environment isn't the simplest, but hopefully this is the last time we have to change the codebase to this extent. I'll update the wiki for new developers later; this is meant for those of us dealing with the transition.

Yes, this e-mail is long, even if you've come to expect that from me. But please make sure you read this one. :)


= Migrate =

You're going to lose everything in this process, but ultimately it's a dev environment, so you probably don't have much in there to begin with.

The first steps are going to be to remove the old Pulp dev install. From what I hear, there's issues in pulp-dev's uninstall capabilities when dealing with symlinks, so for simplicity we'll do a few manual steps:

1. $ sudo systemctl stop httpd.service
2. $ sudo rm -rf /etc/pulp/*
3. $ sudo rm -rf /var/lib/pulp/*
4. $ sudo rm /usr/bin/pulp-*
5. Drop your database entirely:
   $ mongo
   $ use pulp_database
   $ db.dropDatabase()

Now for the new stuff. If you don't have any outstanding code, it might be quicker to just move your code out to a temp location and re-clone the repo. It will save you from having old stuff lying around.

1. $ git pull

The next few steps are to clear out things that git doesn't know about, such as .pyc files. The following should be run from the git checkout root. To be clear: do not do this if you have outstanding changes to merge in. That should only be prad/john with plugin changes but everyone think before you run these (all of this code has moved and you'll need to merge your changes in elsewhere, more on that later).

2. $ rm -rf src  # yes, you're deleting src in root; it's moved
3. $ rm -rf plugins
4. $ rm -rf extensions

Install the dev environment. You'll need to hit both setup.py files in platform and rpm_support to get all of the site-packages references created.

5. $ sudo python ./pulp-dev.py -I
6. $ cd platform/src && sudo python ./setup.py develop
7. $ cd rpm-support/src && sudo python ./setup.py develop

At this point, things should largely be back to normal:

8. $ sudo pulp-migrate # note the version is back to 1 but I'll address that soon

I found it easier to clean up my logs so I knew of any new issues coming up:

9a. (optional) $ rm -f /var/log/httpd/*
9b. (optional) $ rm -f /var/log/pulp/*

10. $ sudo systemctl start httpd.service


= Validation =

Keep an eye on /var/log/httpd/error_log to make sure the server starts correctly. It won't if the plugins/type definitions aren't happy. If it doesn't, ping me, but also check:

/var/lib/pulp/plugins/{distributors,importers,types}

to make sure there aren't any symlinks in there that don't point to actual files.


= Running the Client =

Apache's mod_ssl seems to have changed a bit in Fedora 17. It used to create a default cert with the CN "localhost.localdomain". Now, at least on my box, the CN is the hostname of the box itself.

That's going to cause a problem when trying to run the client. The host in admin.conf needs to match the CN. We ship with localhost.localdomain by default but that's not sufficient anymore.

That means you'll need to change the host in /etc/pulp/admin/admin.conf in order to do anything. The problem is that that file is symlinked into the codebase, so if you edit it directly you may accidentally commit it.

If you create ~/.pulp/admin.conf it will use that as overrides (~/.pulp/consumer.conf is the corresponding consumer override, but I may remove that since it doesn't really make sense to scope anything consumer related to a specific user). The contents should be:

[server]
host = sunflower
# always makes me smile that sayli named her desktop "sunflower" :)


= Where is Everything? =

I made a slight change to what was proposed last week. The bulk of our codebase is organized into three subprojects:

platform - All of the plumbing, including server, client, and agent. If you ever find yourself mentioning rpm/errata/kickstart/puppet/other_type in here, it's wrong.

rpm-support - RPM pluigns/extensions/handlers. If you ever find yourself mentioning rpm/errata/kickstart _outside_ of this directory, it's wrong.

builtins - The generic extensions/plugins/handlers that ship as part of the Pulp platform. These cover things unrelated to types, such as the create repo that doesn't imply an importer or simple login/logout functionality.

In the future, we'll be adding subprojects for ISO support and puppet support. Maybe even Windows support if Todd holds my children ransom and forces me to.

Within each of the three subprojects there are a few consistent pieces:

src - Source code.
test/unit - Source code for unit tests
test/unit/data - Data needed for unit tests.
etc - Mimics the structure where any files are deployed.

For the plugin-related subprojects, you'll also find:

extensions
handlers
plugins/{distributors,importers,profilers,types}


= Tests =

What the above means is that we have test cases in three different places now (one per subproject). This is actually a benefit in many ways. During development in a particular subproject it's dirt simple and quick as hell to run just the tests for that subproject. We're in a much better position for test-driven development practices; we won't grow old while running the full suite as a sanity check.

I'm serious about quicker. The platform tests run in 60-70 seconds. RPM support runs in around 90. And after stripping out all of the untested v1 code, we're floating around 71% total project code coverage. That's actually pretty damn awesome. It's not gonna stop me from bitching that we need to raise that number, but I do acknowledge that it's a very healthy start.

Back to the three locations. There's a script in root called run-tests.sh that runs the tests in all three locations and generates a coverage report. The coverage report is under cover in the git root (pull up index.html in there for the slick HTML view with the colored markers of uncovered code).

That said, don't run that script yet. There's currently an issue in the RPM support tests that cause them to barf a ton of directories into your working directory and not clean them up, so running that script results in a bunch of garbage thrown into your git checkout. We'll get those fixed ASAP.

Something else to keep in mind. That script will run all three suites in the same nosetests process. Jenkins, however, will intentionally run them in three separate shells. My fear is that the single process run will allow dependencies to bleed across subprojects and not be detected. But at the same time, I dig the full project coverage report. So jenkins will be our safety net for making sure the process divisions are in place between tests in case you ever hit an issue where jenkins indicates a failure but the run-tests.sh script does not.

More on testing in a future e-mail; I have a bunch of comments about things we've done wrong in the past and how the new base classes work.


= Random Notes =

* pulp-consumer is broken right now. There are a few bugs in play. It's always been using the admin certificate instead of its consumer certificate, and when I fixed that it's now failing authentication. A quick look at the code makes me think the auth code isn't looking at the v2 consumers collection, but I really didn't look closely. I'll be addressing this later today.

* I stripped out anything not currently being used. We still have it in git history if we need to look again, but the goal here was to be as clean as possible. That means things related to CDS, auditing, event sending, the exporter, and the recent package migration script are all deleted. We can always look up the v1 implementations when we go to address them in v2 but I'd rather not have such a large sweeping clean up again so took care of them now.

* Jeff's still working on the .spec files. There will be two: one for Pulp platform and one for RPM support, each producing a number of RPMs. I've seen the work done so far and they look great; much better organization by created RPM rather than by spec section. Not sure of his current status but I suspect we'll be building by the end of the week.


I think that's all I have for now. Ping me if you run into any problems with the migration. I'm thinking of setting up an elluminate for the whole day that I'll just sit in and people can jump in/screen share when they run into specific problems, but we'll see how it goes.

--
Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]