[Pulp-list] New Source Layout

[Jay Dobies]

On 06/13/2012 02:12 PM, James Slagle wrote:
> pulp-dev.py creates /usr/lib/pulp/agent/handlers,
> but then tries to install symlinks into /usr/lib/pulp/agent/handler (lack of
> 's').
>
> I'm not sure which is right.  I assume it's working for everyone b/c the
> handler dir is still around from previous setups, but I'm actually trying in a
> pristine environment, so it's failing for me.

Transitional thing. It was changed to handlers but must have missed a 
reference. I'll fix it in a few minutes, about to send out an email 
about some other changes I'm making.


>
> On Wed, Jun 13, 2012 at 09:03:13AM -0400, Jay Dobies wrote:
>> I think we're in a good place to get everyone back on master. The
>> migration process for your dev environment isn't the simplest, but
>> hopefully this is the last time we have to change the codebase to
>> this extent. I'll update the wiki for new developers later; this is
>> meant for those of us dealing with the transition.
>>
>> Yes, this e-mail is long, even if you've come to expect that from
>> me. But please make sure you read this one.  :)
>>
>>
>> = Migrate =
>>
>> You're going to lose everything in this process, but ultimately it's
>> a dev environment, so you probably don't have much in there to begin
>> with.
>>
>> The first steps are going to be to remove the old Pulp dev install.
>>  From what I hear, there's issues in pulp-dev's uninstall
>> capabilities when dealing with symlinks, so for simplicity we'll do
>> a few manual steps:
>>
>> 1. $ sudo systemctl stop httpd.service
>> 2. $ sudo rm -rf /etc/pulp/*
>> 3. $ sudo rm -rf /var/lib/pulp/*
>> 4. $ sudo rm /usr/bin/pulp-*
>> 5. Drop your database entirely:
>>     $ mongo
>>     $ use pulp_database
>>     $ db.dropDatabase()
>>
>> Now for the new stuff. If you don't have any outstanding code, it
>> might be quicker to just move your code out to a temp location and
>> re-clone the repo. It will save you from having old stuff lying
>> around.
>>
>> 1. $ git pull
>>
>> The next few steps are to clear out things that git doesn't know
>> about, such as .pyc files. The following should be run from the git
>> checkout root. To be clear: do not do this if you have outstanding
>> changes to merge in. That should only be prad/john with plugin
>> changes but everyone think before you run these (all of this code
>> has moved and you'll need to merge your changes in elsewhere, more
>> on that later).
>>
>> 2. $ rm -rf src  # yes, you're deleting src in root; it's moved
>> 3. $ rm -rf plugins
>> 4. $ rm -rf extensions
>>
>> Install the dev environment. You'll need to hit both setup.py files
>> in platform and rpm_support to get all of the site-packages
>> references created.
>>
>> 5. $ sudo python ./pulp-dev.py -I
>> 6. $ cd platform/src&&  sudo python ./setup.py develop
>> 7. $ cd rpm-support/src&&  sudo python ./setup.py develop
>>
>> At this point, things should largely be back to normal:
>>
>> 8. $ sudo pulp-migrate  # note the version is back to 1 but I'll
>> address that soon
>>
>> I found it easier to clean up my logs so I knew of any new issues coming up:
>>
>> 9a. (optional) $ rm -f /var/log/httpd/*
>> 9b. (optional) $ rm -f /var/log/pulp/*
>>
>> 10. $ sudo systemctl start httpd.service
>>
>>
>> = Validation =
>>
>> Keep an eye on /var/log/httpd/error_log to make sure the server
>> starts correctly. It won't if the plugins/type definitions aren't
>> happy. If it doesn't, ping me, but also check:
>>
>> /var/lib/pulp/plugins/{distributors,importers,types}
>>
>> to make sure there aren't any symlinks in there that don't point to
>> actual files.
>>
>>
>> = Running the Client =
>>
>> Apache's mod_ssl seems to have changed a bit in Fedora 17. It used
>> to create a default cert with the CN "localhost.localdomain". Now,
>> at least on my box, the CN is the hostname of the box itself.
>>
>> That's going to cause a problem when trying to run the client. The
>> host in admin.conf needs to match the CN. We ship with
>> localhost.localdomain by default but that's not sufficient anymore.
>>
>> That means you'll need to change the host in
>> /etc/pulp/admin/admin.conf in order to do anything. The problem is
>> that that file is symlinked into the codebase, so if you edit it
>> directly you may accidentally commit it.
>>
>> If you create ~/.pulp/admin.conf it will use that as overrides
>> (~/.pulp/consumer.conf is the corresponding consumer override, but I
>> may remove that since it doesn't really make sense to scope anything
>> consumer related to a specific user). The contents should be:
>>
>> [server]
>> host = sunflower
>> # always makes me smile that sayli named her desktop "sunflower" :)
>>
>>
>> = Where is Everything? =
>>
>> I made a slight change to what was proposed last week. The bulk of
>> our codebase is organized into three subprojects:
>>
>> platform - All of the plumbing, including server, client, and agent.
>> If you ever find yourself mentioning
>> rpm/errata/kickstart/puppet/other_type in here, it's wrong.
>>
>> rpm-support - RPM pluigns/extensions/handlers. If you ever find
>> yourself mentioning rpm/errata/kickstart _outside_ of this
>> directory, it's wrong.
>>
>> builtins - The generic extensions/plugins/handlers that ship as part
>> of the Pulp platform. These cover things unrelated to types, such as
>> the create repo that doesn't imply an importer or simple
>> login/logout functionality.
>>
>> In the future, we'll be adding subprojects for ISO support and
>> puppet support. Maybe even Windows support if Todd holds my children
>> ransom and forces me to.
>>
>> Within each of the three subprojects there are a few consistent pieces:
>>
>> src - Source code.
>> test/unit - Source code for unit tests
>> test/unit/data - Data needed for unit tests.
>> etc - Mimics the structure where any files are deployed.
>>
>> For the plugin-related subprojects, you'll also find:
>>
>> extensions
>> handlers
>> plugins/{distributors,importers,profilers,types}
>>
>>
>> = Tests =
>>
>> What the above means is that we have test cases in three different
>> places now (one per subproject). This is actually a benefit in many
>> ways. During development in a particular subproject it's dirt simple
>> and quick as hell to run just the tests for that subproject. We're
>> in a much better position for test-driven development practices; we
>> won't grow old while running the full suite as a sanity check.
>>
>> I'm serious about quicker. The platform tests run in 60-70 seconds.
>> RPM support runs in around 90. And after stripping out all of the
>> untested v1 code, we're floating around 71% total project code
>> coverage. That's actually pretty damn awesome. It's not gonna stop
>> me from bitching that we need to raise that number, but I do
>> acknowledge that it's a very healthy start.
>>
>> Back to the three locations. There's a script in root called
>> run-tests.sh that runs the tests in all three locations and
>> generates a coverage report. The coverage report is under cover in
>> the git root (pull up index.html in there for the slick HTML view
>> with the colored markers of uncovered code).
>>
>> That said, don't run that script yet. There's currently an issue in
>> the RPM support tests that cause them to barf a ton of directories
>> into your working directory and not clean them up, so running that
>> script results in a bunch of garbage thrown into your git checkout.
>> We'll get those fixed ASAP.
>>
>> Something else to keep in mind. That script will run all three
>> suites in the same nosetests process. Jenkins, however, will
>> intentionally run them in three separate shells. My fear is that the
>> single process run will allow dependencies to bleed across
>> subprojects and not be detected. But at the same time, I dig the
>> full project coverage report. So jenkins will be our safety net for
>> making sure the process divisions are in place between tests in case
>> you ever hit an issue where jenkins indicates a failure but the
>> run-tests.sh script does not.
>>
>> More on testing in a future e-mail; I have a bunch of comments about
>> things we've done wrong in the past and how the new base classes
>> work.
>>
>>
>> = Random Notes =
>>
>> * pulp-consumer is broken right now. There are a few bugs in play.
>> It's always been using the admin certificate instead of its consumer
>> certificate, and when I fixed that it's now failing authentication.
>> A quick look at the code makes me think the auth code isn't looking
>> at the v2 consumers collection, but I really didn't look closely.
>> I'll be addressing this later today.
>>
>> * I stripped out anything not currently being used. We still have it
>> in git history if we need to look again, but the goal here was to be
>> as clean as possible. That means things related to CDS, auditing,
>> event sending, the exporter, and the recent package migration script
>> are all deleted. We can always look up the v1 implementations when
>> we go to address them in v2 but I'd rather not have such a large
>> sweeping clean up again so took care of them now.
>>
>> * Jeff's still working on the .spec files. There will be two: one
>> for Pulp platform and one for RPM support, each producing a number
>> of RPMs. I've seen the work done so far and they look great; much
>> better organization by created RPM rather than by spec section. Not
>> sure of his current status but I suspect we'll be building by the
>> end of the week.
>>
>>
>> I think that's all I have for now. Ping me if you run into any
>> problems with the migration. I'm thinking of setting up an
>> elluminate for the whole day that I'll just sit in and people can
>> jump in/screen share when they run into specific problems, but we'll
>> see how it goes.
>>
>> --
>> Jay Dobies
>> Freenode: jdob @ #pulp
>> http://pulpproject.org | http://blog.pulpproject.org
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
> --
> -- James Slagle
> --


-- 
Jay Dobies
Freenode: jdob @ #pulp
http://pulpproject.org | http://blog.pulpproject.org