[Pulp-list] pulp-manage-db freezes on fresh installation?

Wed Jun 24 13:33:04 UTC 2015

Ben,

Is this a machine that had previously had Pulp installed? I'm wondering if there were old /var/lib/pulp/published/yum/master/* directories that it still had to scan? Migration 20 is scanning that directory and modifying the contents. That should not be particularly intensive unless there is something going on with the filesystem. Regards,

-Barnaby

----- Original Message -----
From: "ben stanley" <ben.stanley at exemail.com.au>
To: "Ben Stanley" <Ben.Stanley at exemail.com.au>
Cc: "Barnaby Court" <bcourt at redhat.com>, pulp-list at redhat.com
Sent: Wednesday, June 24, 2015 1:37:31 AM
Subject: Re: [Pulp-list] pulp-manage-db freezes on fresh installation?

The machine with the problem was shut down yesterday.

I tried pulp-manage-db again today and it worked.

-----------------------------------------------------------------------------
[root at stig scripts]# sudo -u apache pulp-manage-db
Mongo database for connection is version 2.6.9
Loading content types.
Loading type descriptors [iso_support.json, rpm_support.json, puppet.json]
Parsing type descriptors
Validating type descriptor syntactic integrity
Validating type descriptor semantic integrity
Updating the database with types [iso, distribution, drpm, erratum,
package_group, package_category, package_environment, rpm, srpm,
yum_repo_metadata_file, puppet_module]
Content types loaded.
Ensuring the admin role and user are in place.
Admin role and user are in place.
Beginning database migrations.
Migration package pulp.server.db.migrations is up to date at version 12
Migration package pulp_puppet.plugins.migrations is up to date at version 2
Applying pulp_rpm.plugins.migrations version 20
Migration to pulp_rpm.plugins.migrations version 20 complete.
Applying pulp_rpm.plugins.migrations version 21
Migration to pulp_rpm.plugins.migrations version 21 complete.
Database migrations complete.
[root at stig scripts]#
-----------------------------------------------------------------------------

It took a while to do version 20. I walked away for an hour. When I got
back it was done. I only waited 10min the first time. Perhaps it wasn't
enough?

I am wondering why it takes a long time to migrate a database version when
there are no repos on this machine. It is a clean install.

Ben.

> Barnaby,
>
> Thanks for your reply.
>
> This is the secondary pulp system that I am trying to set up. There are NO
> repositories on it yet (as I un-installed pulp and deleted /var/lib/pulp
> and /var/lib/mongodb and then re-installed). It was while I was
> re-installing that the update-db problem occurred.
>
> I am having a lot of trouble establishing a disconnected sync method, but
> that is documented on plan.io . I look forward to you finding that...
>
> Ben.
>
>
> On 22 June 2015 11:06:30 pm Barnaby Court <bcourt at redhat.com> wrote:
>
>> Hi Ben,
>>
>> How many published repositories do you have and how fast is their
>> filesystem?  pulp_rpm migration 20 is fixing an issue with the directory
>> structure created for published delta rpm files. If you have many
>> published
>> repositories on a slow filesystem this could take a while as it could be
>> removing a lot of bad symlinks.
>>
>> -Barnaby
>>
>> ----- Original Message -----
>> From: "ben stanley" <ben.stanley at exemail.com.au>
>> To: pulp-list at redhat.com
>> Sent: Monday, June 22, 2015 1:59:33 AM
>> Subject: [Pulp-list] pulp-manage-db freezes on fresh installation?
>>
>> I had some trouble running pulp-manage-db on a fresh installation.
>>
>> The update process locks up at update 20.
>>
>> I have provided a script of what I did (attached pulp_installation.txt).
>> I can reproduce it by removing all the packages, cleaning
>> out /var/lib/mongodb, and running the pulp installation again from the
>> top.
>>
>> I looked in the log, but there wasn't anything illuminating.
>>
>> What could be causing this?
>>
>> Ben.
>>
>> ------------------------ /var/log/messages -----------------------------
>> pulp: pulp.server.db.manage:INFO: Migration to
>> pulp_rpm.plugins.migrations version 16 complete.
>> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
>> version 17
>> pulp: pulp.server.db.manage:INFO: Migration to
>> pulp_rpm.plugins.migrations version 17 complete.
>> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
>> version 18
>> pulp: pulp.server.db.manage:INFO: Migration to
>> pulp_rpm.plugins.migrations version 18 complete.
>> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
>> version 19
>> pulp: pulp.server.db.manage:INFO: Migration to
>> pulp_rpm.plugins.migrations version 19 complete.
>> pulp: pulp.server.db.manage:INFO: Applying pulp_rpm.plugins.migrations
>> version 20
>> ------------------------------------------------------------------------
>>
>> ---------------- Output of just the pulp-manage-db ---------------------
>> [root at stig lib]# sudo -u apache pulp-manage-db
>> Mongo database for connection is version 2.6.9
>> Loading content types.
>> Loading type descriptors [iso_support.json, rpm_support.json,
>> puppet.json]
>> Parsing type descriptors
>> Validating type descriptor syntactic integrity
>> Validating type descriptor semantic integrity
>> Updating the database with types [iso, distribution, drpm, erratum,
>> package_group, package_category, package_environment, rpm, srpm,
>> yum_repo_metadata_file, puppet_module]
>> Content types loaded.
>> Ensuring the admin role and user are in place.
>> Admin role and user are in place.
>> Beginning database migrations.
>> Applying pulp.server.db.migrations version 1
>> Migration to pulp.server.db.migrations version 1 complete.
>> Applying pulp.server.db.migrations version 2
>> Migration to pulp.server.db.migrations version 2 complete.
>> Applying pulp.server.db.migrations version 3
>> Migration to pulp.server.db.migrations version 3 complete.
>> Applying pulp.server.db.migrations version 4
>> regenerating content unit counts for 0 repositories
>> Migration to pulp.server.db.migrations version 4 complete.
>> Applying pulp.server.db.migrations version 5
>> Migration to pulp.server.db.migrations version 5 complete.
>> Applying pulp.server.db.migrations version 6
>> Migration to pulp.server.db.migrations version 6 complete.
>> Applying pulp.server.db.migrations version 7
>> Migration to pulp.server.db.migrations version 7 complete.
>> Applying pulp.server.db.migrations version 8
>> Migration to pulp.server.db.migrations version 8 complete.
>> Applying pulp.server.db.migrations version 9
>> Migration to pulp.server.db.migrations version 9 complete.
>> Applying pulp.server.db.migrations version 10
>> Migration to pulp.server.db.migrations version 10 complete.
>> Applying pulp.server.db.migrations version 11
>> Migration to pulp.server.db.migrations version 11 complete.
>> Applying pulp.server.db.migrations version 12
>> Migration to pulp.server.db.migrations version 12 complete.
>> Applying pulp_puppet.plugins.migrations version 1
>> Migrated puppet modules to include checksum
>> Migration to pulp_puppet.plugins.migrations version 1 complete.
>> Applying pulp_puppet.plugins.migrations version 2
>> Migration to pulp_puppet.plugins.migrations version 2 complete.
>> Applying pulp_rpm.plugins.migrations version 1
>> Export distributor migration for rpm repositories started
>> Export distributor migration for rpm repositories finished
>> Migration to pulp_rpm.plugins.migrations version 1 complete.
>> Applying pulp_rpm.plugins.migrations version 2
>> Migrated rpms to include rpm metadata
>> Migration to pulp_rpm.plugins.migrations version 2 complete.
>> Applying pulp_rpm.plugins.migrations version 3
>> Migration to pulp_rpm.plugins.migrations version 3 complete.
>> Applying pulp_rpm.plugins.migrations version 4
>> Migration to pulp_rpm.plugins.migrations version 4 complete.
>> Applying pulp_rpm.plugins.migrations version 5
>> Migrated rpms to include rpm changelog and filelist metadata
>> Migration to pulp_rpm.plugins.migrations version 5 complete.
>> Applying pulp_rpm.plugins.migrations version 6
>> Migration to pulp_rpm.plugins.migrations version 6 complete.
>> Applying pulp_rpm.plugins.migrations version 7
>> No yum repositories found to inventory custom metadata on
>> Migration to pulp_rpm.plugins.migrations version 7 complete.
>> Applying pulp_rpm.plugins.migrations version 8
>> Migration to pulp_rpm.plugins.migrations version 8 complete.
>> Applying pulp_rpm.plugins.migrations version 9
>> Migration to pulp_rpm.plugins.migrations version 9 complete.
>> Applying pulp_rpm.plugins.migrations version 10
>> Migration to pulp_rpm.plugins.migrations version 10 complete.
>> Applying pulp_rpm.plugins.migrations version 11
>> Migration to pulp_rpm.plugins.migrations version 11 complete.
>> Applying pulp_rpm.plugins.migrations version 12
>> Migration to pulp_rpm.plugins.migrations version 12 complete.
>> Applying pulp_rpm.plugins.migrations version 13
>> Migration to pulp_rpm.plugins.migrations version 13 complete.
>> Applying pulp_rpm.plugins.migrations version 14
>> Migration to pulp_rpm.plugins.migrations version 14 complete.
>> Applying pulp_rpm.plugins.migrations version 15
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
>> Client-5.7-i386/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
>> Client-5.7-x86_64/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
>> Client-5.8-i386/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise Linux
>> Client-5.8-x86_64/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.1-i386/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.1-x86_64/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.2-i386/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.2-x86_64/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.3-i386/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.3-x86_64/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.4-i386/treeinfo
>> stripping repomd.xml checksum
>> from /var/lib/pulp/content/distribution/ks-Red Hat Enterprise
>> Linux-Workstation-6.4-x86_64/treeinfo
>> examining distribution units
>> Migration to pulp_rpm.plugins.migrations version 15 complete.
>> Applying pulp_rpm.plugins.migrations version 16
>> Loaded plugin yum_distributor for types:
>> rpm,srpm,drpm,erratum,package_group,package_category,distribution,yum_repo_metadata_file
>> Loaded plugin export_distributor for types:
>> rpm,srpm,drpm,erratum,distribution,package_category,package_group
>> Loaded plugin iso_distributor for types: iso
>> Loaded plugin puppet_distributor for types: puppet_module
>> Loaded plugin puppet_file_distributor for types: puppet_module
>> Loaded plugin puppet_install_distributor for types: puppet_module
>> Loaded plugin group_export_distributor for types:
>> rpm,srpm,drpm,erratum,distribution,package_category,package_group
>> Loaded plugin yum_importer for types:
>> distribution,drpm,erratum,package_group,package_category,rpm,srpm,yum_repo_metadata_file,package_environment
>> Loaded plugin iso_importer for types: iso
>> Loaded plugin puppet_importer for types: puppet_module
>> Loaded plugin yum_profiler for types: rpm,erratum
>> Loaded plugin puppet_whole_repo_profiler for types: puppet_module
>> Loaded plugin yum for types: rpm
>> Loaded plugin rhui for types: rpm
>> Migration to pulp_rpm.plugins.migrations version 16 complete.
>> Applying pulp_rpm.plugins.migrations version 17
>> Migration to pulp_rpm.plugins.migrations version 17 complete.
>> Applying pulp_rpm.plugins.migrations version 18
>> Migration to pulp_rpm.plugins.migrations version 18 complete.
>> Applying pulp_rpm.plugins.migrations version 19
>> Migration to pulp_rpm.plugins.migrations version 19 complete.
>> Applying pulp_rpm.plugins.migrations version 20
>>
>> <waited 10min>
>> ------------------------------------------------------------------------
>>
>>
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
>