[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]

Re: [Pulp-list] New Source Layout

On 06/13/2012 02:12 PM, James Slagle wrote:
pulp-dev.py creates /usr/lib/pulp/agent/handlers,
but then tries to install symlinks into /usr/lib/pulp/agent/handler (lack of

I'm not sure which is right.  I assume it's working for everyone b/c the
handler dir is still around from previous setups, but I'm actually trying in a
pristine environment, so it's failing for me.

Transitional thing. It was changed to handlers but must have missed a reference. I'll fix it in a few minutes, about to send out an email about some other changes I'm making.

On Wed, Jun 13, 2012 at 09:03:13AM -0400, Jay Dobies wrote:
I think we're in a good place to get everyone back on master. The
migration process for your dev environment isn't the simplest, but
hopefully this is the last time we have to change the codebase to
this extent. I'll update the wiki for new developers later; this is
meant for those of us dealing with the transition.

Yes, this e-mail is long, even if you've come to expect that from
me. But please make sure you read this one.  :)

= Migrate =

You're going to lose everything in this process, but ultimately it's
a dev environment, so you probably don't have much in there to begin

The first steps are going to be to remove the old Pulp dev install.
 From what I hear, there's issues in pulp-dev's uninstall
capabilities when dealing with symlinks, so for simplicity we'll do
a few manual steps:

1. $ sudo systemctl stop httpd.service
2. $ sudo rm -rf /etc/pulp/*
3. $ sudo rm -rf /var/lib/pulp/*
4. $ sudo rm /usr/bin/pulp-*
5. Drop your database entirely:
    $ mongo
    $ use pulp_database
    $ db.dropDatabase()

Now for the new stuff. If you don't have any outstanding code, it
might be quicker to just move your code out to a temp location and
re-clone the repo. It will save you from having old stuff lying

1. $ git pull

The next few steps are to clear out things that git doesn't know
about, such as .pyc files. The following should be run from the git
checkout root. To be clear: do not do this if you have outstanding
changes to merge in. That should only be prad/john with plugin
changes but everyone think before you run these (all of this code
has moved and you'll need to merge your changes in elsewhere, more
on that later).

2. $ rm -rf src  # yes, you're deleting src in root; it's moved
3. $ rm -rf plugins
4. $ rm -rf extensions

Install the dev environment. You'll need to hit both setup.py files
in platform and rpm_support to get all of the site-packages
references created.

5. $ sudo python ./pulp-dev.py -I
6. $ cd platform/src&&  sudo python ./setup.py develop
7. $ cd rpm-support/src&&  sudo python ./setup.py develop

At this point, things should largely be back to normal:

8. $ sudo pulp-migrate  # note the version is back to 1 but I'll
address that soon

I found it easier to clean up my logs so I knew of any new issues coming up:

9a. (optional) $ rm -f /var/log/httpd/*
9b. (optional) $ rm -f /var/log/pulp/*

10. $ sudo systemctl start httpd.service

= Validation =

Keep an eye on /var/log/httpd/error_log to make sure the server
starts correctly. It won't if the plugins/type definitions aren't
happy. If it doesn't, ping me, but also check:


to make sure there aren't any symlinks in there that don't point to
actual files.

= Running the Client =

Apache's mod_ssl seems to have changed a bit in Fedora 17. It used
to create a default cert with the CN "localhost.localdomain". Now,
at least on my box, the CN is the hostname of the box itself.

That's going to cause a problem when trying to run the client. The
host in admin.conf needs to match the CN. We ship with
localhost.localdomain by default but that's not sufficient anymore.

That means you'll need to change the host in
/etc/pulp/admin/admin.conf in order to do anything. The problem is
that that file is symlinked into the codebase, so if you edit it
directly you may accidentally commit it.

If you create ~/.pulp/admin.conf it will use that as overrides
(~/.pulp/consumer.conf is the corresponding consumer override, but I
may remove that since it doesn't really make sense to scope anything
consumer related to a specific user). The contents should be:

host = sunflower
# always makes me smile that sayli named her desktop "sunflower" :)

= Where is Everything? =

I made a slight change to what was proposed last week. The bulk of
our codebase is organized into three subprojects:

platform - All of the plumbing, including server, client, and agent.
If you ever find yourself mentioning
rpm/errata/kickstart/puppet/other_type in here, it's wrong.

rpm-support - RPM pluigns/extensions/handlers. If you ever find
yourself mentioning rpm/errata/kickstart _outside_ of this
directory, it's wrong.

builtins - The generic extensions/plugins/handlers that ship as part
of the Pulp platform. These cover things unrelated to types, such as
the create repo that doesn't imply an importer or simple
login/logout functionality.

In the future, we'll be adding subprojects for ISO support and
puppet support. Maybe even Windows support if Todd holds my children
ransom and forces me to.

Within each of the three subprojects there are a few consistent pieces:

src - Source code.
test/unit - Source code for unit tests
test/unit/data - Data needed for unit tests.
etc - Mimics the structure where any files are deployed.

For the plugin-related subprojects, you'll also find:


= Tests =

What the above means is that we have test cases in three different
places now (one per subproject). This is actually a benefit in many
ways. During development in a particular subproject it's dirt simple
and quick as hell to run just the tests for that subproject. We're
in a much better position for test-driven development practices; we
won't grow old while running the full suite as a sanity check.

I'm serious about quicker. The platform tests run in 60-70 seconds.
RPM support runs in around 90. And after stripping out all of the
untested v1 code, we're floating around 71% total project code
coverage. That's actually pretty damn awesome. It's not gonna stop
me from bitching that we need to raise that number, but I do
acknowledge that it's a very healthy start.

Back to the three locations. There's a script in root called
run-tests.sh that runs the tests in all three locations and
generates a coverage report. The coverage report is under cover in
the git root (pull up index.html in there for the slick HTML view
with the colored markers of uncovered code).

That said, don't run that script yet. There's currently an issue in
the RPM support tests that cause them to barf a ton of directories
into your working directory and not clean them up, so running that
script results in a bunch of garbage thrown into your git checkout.
We'll get those fixed ASAP.

Something else to keep in mind. That script will run all three
suites in the same nosetests process. Jenkins, however, will
intentionally run them in three separate shells. My fear is that the
single process run will allow dependencies to bleed across
subprojects and not be detected. But at the same time, I dig the
full project coverage report. So jenkins will be our safety net for
making sure the process divisions are in place between tests in case
you ever hit an issue where jenkins indicates a failure but the
run-tests.sh script does not.

More on testing in a future e-mail; I have a bunch of comments about
things we've done wrong in the past and how the new base classes

= Random Notes =

* pulp-consumer is broken right now. There are a few bugs in play.
It's always been using the admin certificate instead of its consumer
certificate, and when I fixed that it's now failing authentication.
A quick look at the code makes me think the auth code isn't looking
at the v2 consumers collection, but I really didn't look closely.
I'll be addressing this later today.

* I stripped out anything not currently being used. We still have it
in git history if we need to look again, but the goal here was to be
as clean as possible. That means things related to CDS, auditing,
event sending, the exporter, and the recent package migration script
are all deleted. We can always look up the v1 implementations when
we go to address them in v2 but I'd rather not have such a large
sweeping clean up again so took care of them now.

* Jeff's still working on the .spec files. There will be two: one
for Pulp platform and one for RPM support, each producing a number
of RPMs. I've seen the work done so far and they look great; much
better organization by created RPM rather than by spec section. Not
sure of his current status but I suspect we'll be building by the
end of the week.

I think that's all I have for now. Ping me if you run into any
problems with the migration. I'm thinking of setting up an
elluminate for the whole day that I'll just sit in and people can
jump in/screen share when they run into specific problems, but we'll
see how it goes.

Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org

Pulp-list mailing list
Pulp-list redhat com
-- James Slagle

Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]