[Pulp-list] Repo configured to use SHA checksums, but uploaded packages still get SHA256
Philipp Gassmann
gassmann at puzzle.ch
Tue Jun 17 15:33:49 UTC 2014
Hi,
I tried no to modify the
/usr/lib/python2.6/site-packages/pulp_rpm/extension/admin/upload/package.py
as suggested by Christina Plummer.
(on the separate build-server, not on the pulp server)
just replacing sha256 with sha didn't work.
_checksum_calculate_checksum(type,file) needs 'sha1' as type to generate
a correct sha1 hash, but the checksum type for the pulp-metadata needs
to be 'sha'.
Now I managed to get the pulp-metadata right when i check on the command
line:
# pulp-admin rpm repo content rpm --repo-id centos.5.x86_64.puzzle |
grep Data-Random -C 20
Arch: noarch
Buildhost: buildserver.not.puzzle.ch
Checksum: 4ad4b6eb87df53002f2c25223495a1503458420f
Checksumtype: sha
Description: A module used to generate random data. Useful mostly for test
programs.
Epoch: 0
Filename: perl-Data-Random-0.05-1.el5.noarch.rpm
License: Artistic/GPL
Name: perl-Data-Random
Provides: perl(Data::Random) = 0.05-0, perl(Data::Random::WordList) =
0.05-0, perl-Data-Random = 0.05-1.el5-0
Release: 1.el5
Requires: perl, perl(Carp), perl(Date::Calc), perl(Exporter), perl(GD)
Vendor: Dag Apt Repository, http://dag.wieers.com/apt/
Version: 0.05
BUT then I run the repo publish command to generate the xml metadata
# pulp-admin rpm repo publish run --repo-id centos.5.x86_64.puzzle
+----------------------------------------------------------------------+
Publishing Repository [centos.5.x86_64.puzzle]
+----------------------------------------------------------------------+
This command may be exited by pressing ctrl+c without affecting the actual
operation on the server.
Publishing distributions...
[==================================================] 100%
Distributions: 0/0 items
... completed
Publishing packages...
[==================================================] 100%
Packages: 157/157 items
... completed
Generating metadata
[\]
... completed
Publishing repository over HTTPS
[-]
... completed
Publishing repository over HTTP
[\]
... completed
AND it still generates the WRONG Hash to the repodata/primary.xml.gz:
<package type="rpm">
<name>perl-Data-Random</name>
<arch>noarch</arch><version epoch="0" ver="0.05" rel="1.el5"/>
<checksum type="sha256"
pkgid="YES">c45114bb7c1b2331e1bc352a691d3eaac1a14f3a6a5011eeae73c1eb8c8f86a2</checksum>
...
It looks completely broken, inconsistent on various levels
(architecture/implementation/process).
- checksum calculated per upload
- checksum type per file in a repository
- checksum type as repository option (OK) but without effect.
- inconsistent checksum (type) between metadata in database and in xml
repodata
- type sha means hashlib.new('sha1')
- Manual Package upload should be a core functionality of pulp and it is
so broken
- Bug is known since September last year
- Bug is categorized to "user-experience"
- Pulp is supposed to be a core part of Satellite 6
Sorry for the rant, but I expect better quality from Red Hat.
We want to use pulp as yum repository mirror and Server for RHEL/CentOS
5 & 6, stumbling over such a blocker is very disappointing.
Regards,
Philipp Gassmann
Am 17.06.2014 15:36, schrieb Philipp Gassmann:
> Hi Barnaby,
>
> This option (--checksum-type) is not present in the current stable
> version of pulp (2.3.1) as Christina Plummer already mentioned in March.
>
> And as mentioned below, it is still hardcoded in the current master branch:
> # Checksum
> unit_key['checksumtype'] = 'sha256' # hardcoded to this in v1 so
> leaving this way for now
>
> https://github.com/pulp/pulp_rpm/blame/master/extensions_admin/pulp_rpm/extensions/admin/upload/package.py#L178
>
>
> from the details of repo list --detail (see below) it looks like
> "Checksum Type" is a property of the distributor of the repository. That
> makes sense to me, as the whole repository is to be consumed by a EL 5
> client.
> BTW. "Auto Publish: True" does not seem to work when manually uploading
> a package (it did in pulp v1).
>
> Regards,
> Philipp
>
> Am 17.06.2014 15:02, schrieb Barnaby Court:
>> Hi, Pulp does support EL5 platforms. The trick is that you have to specify the checksum during the upload of RPM content units. Pulp treats files with a different checksum as completely different entities. Use the "pulp-admin rpm repo uploads rpm --checksum-type sha ..." for uploading the package for RHEL5. This allows you to override the default checksum type of sha256 whenever you upload an RPM. Upload the same rpm twice, once with --checksum-type sha for your el5 repo and once without the option specified or specified as sha256 for your EL6 repo. Regards,
>>
>> Barnaby
>>
>> ----- Original Message -----
>> From: "Philipp Gassmann" <gassmann at puzzle.ch>
>> To: "Martin Collins" <martin.collins at framestore.com>, "Christina Plummer" <cplummer at gmail.com>, "Barnaby Court" <bcourt at redhat.com>
>> Cc: pulp-list at redhat.com
>> Sent: Tuesday, June 17, 2014 4:51:57 AM
>> Subject: Re: [Pulp-list] Repo configured to use SHA checksums, but uploaded packages still get SHA256
>>
>> Thanks for the replys!
>> Unfortuately those workarounds are not really usable if you need to
>> upload packages to existing repos for EL5 and EL6.
>> I migrated the EL5 repos from pulp_v1 by initially syncing from those
>> repos. Now I need to add additional packages to those repositories.
>>
>> I'm quite disappointed to find out that Pulp 2 does support repositories
>> for EL5, even though RHEL 5 is listed as supported Platform.
>>
>> http://www.pulpproject.org/
>> Supported Platforms:
>> Fedora 18, Fedora 19, Red Hat Enterprise Linux 5, Red Hat Enterprise Linux 6
>>
>> The sha256 is still hardcoded:
>> https://github.com/pulp/pulp_rpm/blob/master/extensions_admin/pulp_rpm/extensions/admin/upload/package.py
>>
>> The checksumtype is a repository option so the checksum should not be
>> different for different packages of the same repository.
>> Pulp 1 had the command to generate the metadata.
>> Pulp 2 has the publish command which has a stage "generating metadata",
>> there the checksum should be created.
>> Am I wrong on that?
>> Why does pulp 2.4 add an option "--checksum-type" to the upload command?
>> Shouln't it use the repository checksum-type option?
>> What is the checksum-type option of the repository for?
>>
>> pulp-admin rpm repo update --repo-id centos.5.x86_64.custom
>> --checksum-type=sha1
>>
>> Id: centos.5.i386.updates
>> Display Name: centos.5.i386.updates
>> Description: Last checked by Puppet on Tue Jun 17 06:43:30 CEST 2014
>> Content Unit Counts:
>> Rpm: 408
>> Notes:
>> Importers:
>> Config:
>> Feed: http://mirror.switch.ch/ftp/mirror/centos/5/updates/i386/
>> Id: yum_importer
>> Importer Type Id: yum_importer
>> Last Sync: 2014-05-21T11:54:49+02:00
>> Repo Id: centos.5.i386.updates
>> Scheduled Syncs:
>> Distributors:
>> Auto Publish: True
>> Config:
>> Checksum Type: sha1
>> Http: True
>> Https: True
>> Relative URL: centos.5.i386.updates
>> Distributor Type Id: yum_distributor
>> Id: yum_distributor
>> Last Publish: 2014-05-21T11:55:24+02:00
>> Repo Id: centos.5.i386.updates
>> Scheduled Publishes:
>>
>>
>> The Bug Report is here:
>> https://bugzilla.redhat.com/show_bug.cgi?id=1004580
>>
>>
>> Am 16.06.2014 23:37, schrieb Martin Collins:
>>> My workaround (as I have a large number of rhel6 and only a few for
>>> rhel5) was to just sync from a file:/// feed URL for rhel5 then just put
>>> the rpms in that dir, createrepo -s sha, then sync and continue normally
>>> from there
>>>
>>> Cheers
>>> Martin
>>>
>>> On 16 Jun 2014 22:33, "Christina Plummer" <cplummer at gmail.com
>>> <mailto:cplummer at gmail.com>> wrote:
>>>
>>> Hi Philipp,
>>>
>>> My workaround was to modify the upload/package.py and explicitly
>>> force it to use "sha" instead of "sha256". On my system, the file
>>> was
>>> /usr/lib/python2.6/site-packages/pulp_rpm/extension/admin/upload/package.py
>>> . See attached diff.
>>>
>>> Be sure to set aside the package.pyc and package.pyo files as well,
>>> and restart httpd. Then delete and re-upload the package to your repos.
>>>
>>> Belated public thanks to Ryan Bowlby for the pointer on this one.
>>>
>>> Christina
>>>
>>>
>>> On Mon, Jun 16, 2014 at 10:19 AM, Philipp Gassmann
>>> <gassmann at puzzle.ch <mailto:gassmann at puzzle.ch>> wrote:
>>>
>>> Hello Christina
>>>
>>> I just stumbled over the same issue. I uploaded the same noarch
>>> package
>>> to multiple repositories. now the installation of the package on
>>> CentOS
>>> 5 fails because of the wrong checksum.
>>>
>>> Did you find a solution or workaround?
>>> I can install the package if I download it manually and install
>>> it with
>>> yum --nogpgcheck.
>>>
>>> Regards,
>>> Philipp
>>>
>>>
>>> Am 11.03.2014 14:26, schrieb Christina Plummer:
>>> > Hi Barnaby,
>>> >
>>> > Thanks for your note. Unfortunately, --checksum-type doesn't
>>> seem to be
>>> > an option for the "pulp-admin rpm repo uploads rpm" command:
>>> >
>>> > CWC\cplummer1 at cbigdc-rhel6test ~ $ pulp-admin rpm repo uploads rpm
>>> > --repo-id=q1-ol5-x86_64 --file
>>> oracle-validated-1.0.0-34.el5.x86_64.rpm
>>> > --checksum-type sha
>>> > Command: rpm
>>> > Description: uploads one or more RPMs into a repository
>>> >
>>> > Available Arguments:
>>> >
>>> > --repo-id - (required) unique identifier; only
>>> alphanumeric, .,
>>> > -, and _
>>> > allowed
>>> > --file, -f - full path to a file to upload; may be
>>> specified multiple
>>> > times for multiple files
>>> > --dir, -d - full path to a directory containing files
>>> to upload;
>>> > may be
>>> > specified multiple times for multiple
>>> directories
>>> > -v - display extra information about the upload
>>> process
>>> > --skip-existing - if specified, RPMs that already exist on
>>> the server
>>> > will not
>>> > be uploaded
>>> > The following options were specified but do not exist on the
>>> command:
>>> > --checksum-type
>>> >
>>> > I am using the RPMs for 2.3.1-1. Shouldn't it default to the
>>> > checksum-type for the repository, if specified?
>>> >
>>> > Thanks,
>>> > Christina
>>> >
>>> >
>>> > On Mon, Mar 10, 2014 at 2:31 PM, Barnaby Court
>>> <bcourt at redhat.com <mailto:bcourt at redhat.com>
>>> > <mailto:bcourt at redhat.com <mailto:bcourt at redhat.com>>> wrote:
>>> >
>>> > Christina,
>>> >
>>> > The checksum on the RPM is set at upload time. Was
>>> "--checksum-type
>>> > SHA1" set on the "pulp-admin rpm repo uploads rpm"
>>> command? If no
>>> > checksum type is specified it defaults to SHA256.
>>> >
>>> > Regards,
>>> > Barnaby
>>> >
>>> > ----- Original Message -----
>>> > From: "Christina Plummer" <cplummer at gmail.com
>>> <mailto:cplummer at gmail.com>
>>> > <mailto:cplummer at gmail.com <mailto:cplummer at gmail.com>>>
>>> > To: pulp-list at redhat.com <mailto:pulp-list at redhat.com>
>>> <mailto:pulp-list at redhat.com <mailto:pulp-list at redhat.com>>
>>> > Sent: Monday, March 10, 2014 1:42:14 PM
>>> > Subject: [Pulp-list] Repo configured to use SHA checksums,
>>> but
>>> > uploaded packages still get SHA256
>>> >
>>> > On Pulp 2.1.3, I had run into issues with sha256 checksums
>>> being
>>> > created for some of the repo metadata files, even though
>>> the repo
>>> > was configured to use sha instead.
>>> >
>>> > On Pulp 2.3.1, this seems to have been fixed... however, I
>>> am now
>>> > seeing that packages manually uploaded into a repo are
>>> still getting
>>> > a sha256 checksum in the repodata, which means that RHEL5/OL5
>>> > clients can't handle them. Packages that were synced, or
>>> copied from
>>> > another repository, have the correct checksum type.
>>> >
>>> > OS: RHEL 6.5 x86_64
>>> > Pulp: 2.3.1-1 (RPMs from Pulp repo)
>>> > Steps to reproduce:
>>> > 1. Created "live" repo fed from Oracle public yum
>>> repositories,
>>> > checksum-type=sha, retain-old-count=1
>>> > 2. Created "q1" repo, checksum-type=sha
>>> > 3. Copied rpms from "live" to "q1
>>> > 4. Deleted "known-broken" versions of
>>> oracle-validated-1.1.0 package
>>> > 5. Uploaded "known-working" version of
>>> oracle-validated-1.0.0 package
>>> > 6. Published repo. Every package except for
>>> oracle-validated-1.0.0
>>> > has a 'sha' checksum; oracle-validated-1.0.0 alone has a
>>> 'sha256'
>>> > checksum.
>>> >
>>> > Is this a bug?
>>> >
>>> > Thanks,
>>> > Christina
>>> >
>>> > _______________________________________________
>>> > Pulp-list mailing list
>>> > Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
>>> <mailto:Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>>
>>> > https://www.redhat.com/mailman/listinfo/pulp-list
>>> >
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > Pulp-list mailing list
>>> > Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
>>> > https://www.redhat.com/mailman/listinfo/pulp-list
>>> >
>>>
>>>
>>> --
>>> Philipp Gassmann
>>>
>>> Puzzle ITC GmbH
>>> http://www.puzzle.ch
>>>
>>> Telefon +41 31 370 22 00 <tel:%2B41%2031%20370%2022%2000>
>>> Werfen Sie einen Blick in unseren Blog:
>>> http://www.puzzle.ch/blog
>>>
>>>
>>>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com <mailto:Pulp-list at redhat.com>
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>
>>
>
>
--
Philipp Gassmann
Puzzle ITC GmbH
http://www.puzzle.ch
Telefon +41 31 370 22 00
Direkt +41 31 370 22 02
Fax +41 31 370 22 01
Werfen Sie einen Blick in unseren Blog:
http://www.puzzle.ch/blog
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 884 bytes
Desc: OpenPGP digital signature
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20140617/bcf72e79/attachment.sig>
More information about the Pulp-list
mailing list