[Pulp-list] Missing packages in clones

[Chris St. Pierre]

We've been tracking a Pulp issue for a while now, but we haven't found
a replication scenario for it just yet.  What happens is that a repo
-- usually a clone or a clone-of-a-clone -- simply starts missing a
few (or a great many) package files, although the packages are listed
in the repo metadata.  There's nothing in either the Pulp or the
Grinder logs to indicate why the packages are missing.

For instance:

% pulp-admin repo content --id=lustre-prod-rhel-6-x86_64-updates | grep audit
audit-viewer-0.5.1-3.el6.x86_64.rpm
audit-2.1.3-3.el6.x86_64.rpm
audit-libs-2.1.3-3.el6.i686.rpm
audit-libs-devel-2.1.3-3.el6.x86_64.rpm
audit-libs-python-2.1.3-3.el6.x86_64.rpm
audit-libs-2.1.3-3.el6.x86_64.rpm
audit-libs-devel-2.1.3-3.el6.i686.rpm

% gunzip -c repodata/primary.xml.gz | grep '<name>audit'
  <name>audit</name>
  <name>audit-libs</name>
  <name>audit-libs-devel</name>
  <name>audit-libs-devel</name>
  <name>audit-libs-python</name>
  <name>audit-viewer</name>
  <name>audit-libs</name>

% ls audit*
audit-viewer-0.5.1-3.el6.x86_64.rpm

It's like audit and audit-libs* have been removed.  They're present in
the parent repo, though:

% pulp-admin repo info --id=lustre-prod-rhel-6-x86_64-updates
[...snip...]
Feed URL                file:///var/lib/pulp//repos/rhel-6-x86_64-updates
[...snip...]

% pulp-admin repo content --id=rhel-6-x86_64-updates | grep audit
audit-viewer-0.5.1-3.el6.x86_64.rpm
audit-2.1.3-3.el6.x86_64.rpm
audit-libs-2.1.3-3.el6.i686.rpm
audit-libs-devel-2.1.3-3.el6.x86_64.rpm
audit-libs-python-2.1.3-3.el6.x86_64.rpm
audit-libs-2.1.3-3.el6.x86_64.rpm
audit-libs-devel-2.1.3-3.el6.i686.rpm

% cd /var/lib/pulp//repos/rhel-6-x86_64-updates
% gunzip -c repodata/primary.xml.gz | grep '<name>audit'
  <name>audit-viewer</name>
  <name>audit</name>
  <name>audit-libs</name>
  <name>audit-libs-devel</name>
  <name>audit-libs-devel</name>
  <name>audit-libs-python</name>
  <name>audit-libs</name>

% ls audit*
audit-2.1.3-3.el6.x86_64.rpm
audit-libs-devel-2.1.3-3.el6.i686.rpm
audit-viewer-0.5.1-3.el6.x86_64.rpm
audit-libs-2.1.3-3.el6.i686.rpm    audit-libs-devel-2.1.3-3.el6.x86_64.rpm
audit-libs-2.1.3-3.el6.x86_64.rpm  audit-libs-python-2.1.3-3.el6.x86_64.rpm

A sync solves the problem -- temporarily.  Without fail, within a few
days packages go missing again.  There doesn't seem to be any rhyme or
reason to which packages go missing; next time it's unlikely that
audit-libs* will be gone again.

It's not always just a few packages, either:

% pulp-admin repo info --id=lustre-prod-rhel-6-x86_64-optional
[...snip...]
Packages                2726
[...snip...]

% ls *.rpm | wc -l
8

This does not seem to happen during a sync; for instance, the
lustre-prod-rhel-6-x86_64-optional repo was synced
2012-02-26T16:35:16-05:00, was working on the 27th of February, and
was broken on the morning of the 28th.  We don't have a good idea of
when exactly the symlinks were removed.

Has anyone else seen this?  Any ideas or suggestions of where to go
next?  Thanks!

-- 
Chris St. Pierre