[Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

[Mike DePaulo]

Hi Joey,

It sounds like aws_default_acl should be documented here then:
https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws

Care to submit a documentation PR?
https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst

Thanks,
-Mike

On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
wrote:

> Turns out the issue was on my end. I had to add aws_default_acl: None to
> the pulp_settings section of the playbook. The public-read ACL was
> incompatible with the BlockPublicAccess settings that I had set on my S3
> bucket. ​
>
> I'm now encountering a different issue, but I'll start another thread for
> that one.
>
> Thanks for the pointers, they were very helpful!
>
> Joey Dumont
>
> Technical Advisor, Knowledge, Information, and Technology Services
> National Research Council Canada / Governement of Canada
> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>
> Conseiller technique, Services du savoir, de l'information et de la
> technologie
> Conseil national de recherches Canada / Gouvernement du Canada
> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436
> ------------------------------
> *From:* David Davis <daviddavis at redhat.com>
> *Sent:* 01 November 2019 15:51
> *To:* Dumont, Joey
> *Cc:* pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running pulp
> on Amazon EC2 (pulp3)
>
> Unfortunately I don't know of a good way to debug the problem other than
> to dig into the code. If you want to debug from the Pulp code, you could
> stick a debugger in the artifact saver stage:
>
>
> https://github.com/pulp/pulpcore/blob/2203fee1407738a4ddd8e644fcbc741aab0bca63/pulpcore/plugin/stages/artifact_stages.py#L179-L200
>
> What I would probably do though is stick a debug statement here in
> django-storages to see what params it's passing to boto3:
>
>
> https://github.com/jschneier/django-storages/blob/0ab2b1e3efd2bcaf0f24540a718993acc7742d9b/storages/backends/s3boto3.py#L511
>
> You can see the location of django-storages with `pip show
> django-storages`.
>
> Sorry I don't have a better answer for you. Perhaps this is something we
> can improve in the future. Also, I'd be curious as to what the issue is as
> it sounds like everything should work in theory.
>
> David
>
>
> On Fri, Nov 1, 2019 at 2:26 PM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
> wrote:
>
>> I've installed the latest pulp3 using the Ansible installer using the
>> following playbook:
>>
>>
>> ---
>> - hosts: mirrors
>>   vars:
>>     prereq_pip_packages:
>>       - django-storages
>>       - boto3
>>     pulp_use_system_wide_pkgs: True
>>     pulp_default_admin_password: !vault |
>>           $ANSIBLE_VAULT;1.1;AES256
>>           ...
>>     pulp_settings:
>>       secret_key: !vault |
>>           $ANSIBLE_VAULT;1.1;AES256
>>           ...
>>       default_file_storage: 'storages.backends.s3boto3.S3Boto3Storage'
>>       aws_storage_bucket_name: 'xxx-pulp-storage'
>>       aws_s3_region_name: 'ca-central-1'
>>       aws_s3_addressing_style: "path"
>>       media_root: '/pulp3/'
>>     pulp_install_plugins:
>>       pulp-file: {}
>>       pulp-rpm:
>>         prereq_role: "pulp.pulp_rpm_prerequisites"
>>         #      pulp-docker: {}
>>   roles:
>>     - pulp-database
>>     - pulp-workers
>>     - pulp-resource-manager
>>     - pulp-webserver
>>     - pulp-content
>>   environment:
>>     DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>
>> I also set up an RPM repo that uses S3 for storage. However, when I try
>> to sync, I get an AccessDenied error. I know the instance profile is
>> correct, as I can upload objects from that instance using both the AWS CLI
>> and Boto3 without specifying credentials.
>>
>> How can I debug this further? Is there a way for me know what parameters
>> are passed to the put_object boto3 call by the sync task?
>>
>> Cheers,
>>
>>
>>
>> Joey Dumont
>>
>> Technical Advisor, Knowledge, Information, and Technology Services
>> National Research Council Canada / Governement of Canada
>> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>
>> Conseiller technique, Services du savoir, de l'information et de la
>> technologie
>> Conseil national de recherches Canada / Gouvernement du Canada
>> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>> 438-340-7436
>> _______________________________________________
>> Pulp-list mailing list
>> Pulp-list at redhat.com
>> https://www.redhat.com/mailman/listinfo/pulp-list
>
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list



-- 

Mike DePaulo

He / Him / His

Service Reliability Engineer, Pulp

Red Hat <https://www.redhat.com/>

IM: mikedep333

GPG: 51745404
<https://www.redhat.com/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20191106/9ff48970/attachment.htm>