[Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)
Mike DePaulo
mikedep333 at redhat.com
Sat Nov 9 21:36:03 UTC 2019
Hmm, I'm not sure if this should be addressed via dynaconf somehow or via
Ansible logic.
For troubleshhooting purposes, we can bypass the Ansible part by modify the
settings.py manually, after it gets written (for example, with a
non-standard Ansible task you add.)
-Mike
On Sat, Nov 9, 2019, 2:05 PM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
wrote:
> I have tried '~', 'null' and '' (without the quotes), and they all get
> rendered as null. This leads to an issue with running the migrations when
> running the Ansible script.
>
>
> I opened an issue: https://pulp.plan.io/issues/5687
>
>
> Maybe this should have gone in the pulp-ansible project?
>
>
> Cheers,
>
>
> Joey Dumont
>
> Technical Advisor, Knowledge, Information, and Technology Services
> National Research Council Canada / Governement of Canada
> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>
> Conseiller technique, Services du savoir, de l'information et de la
> technologie
> Conseil national de recherches Canada / Gouvernement du Canada
> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436
> ------------------------------
> *From:* Mike DePaulo <mikedep333 at redhat.com>
> *Sent:* 09 November 2019 13:47
> *To:* David Davis
> *Cc:* Dumont, Joey; pulp-list at redhat.com
> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running pulp
> on Amazon EC2 (pulp3)
>
> I think that with Ansible syntax, you need to specify:
> AWS_DEFAULT_ACL:
>
> I am fairly certain the lack of any value at the end would get interpreted
> as Python None.
>
> - Mike
>
> On Sat, Nov 9, 2019 at 6:57 AM David Davis <daviddavis at redhat.com> wrote:
>
>> You should be able to set aws_default_acl to null (ie "aws_default_acl:
>> ~" or "aws_default_acl: null"). Sounds like that's not working though. What
>> error did you get? Would you mind filing an issue?
>>
>> https://pulp.plan.io/projects/pulp/issues/new
>>
>> Thanks.
>>
>> David
>>
>>
>> On Fri, Nov 8, 2019 at 10:20 PM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
>> wrote:
>>
>>> I encounted a new issue that escaped my notice by some weird coincidence.
>>>
>>>
>>> It turns that you use the pulp_settings section of the playbook,
>>> aws_default_acl: None gets converted to AWS_DEFAULT_ACL = "None", instead
>>> of AWS_DEFAULT_ACL = None, which results in the cryptic
>>>
>>>
>>> "An error occurred (InvalidArgument) when calling the PutObject
>>> operation: None"
>>>
>>> Is there a way to ensure that Ansible converts aws_default_acl: None to
>>> AWS_DEFAULT_ACL = None?
>>>
>>> I tried setting it to null but I got an error.
>>>
>>> Cheers,
>>>
>>>
>>>
>>> Joey Dumont
>>>
>>> Technical Advisor, Knowledge, Information, and Technology Services
>>> National Research Council Canada / Governement of Canada
>>> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>>
>>> Conseiller technique, Services du savoir, de l'information et de la
>>> technologie
>>> Conseil national de recherches Canada / Gouvernement du Canada
>>> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>>> 438-340-7436
>>> ------------------------------
>>> *From:* pulp-list-bounces at redhat.com <pulp-list-bounces at redhat.com> on
>>> behalf of Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
>>> *Sent:* 06 November 2019 11:36
>>> *To:* Mike DePaulo
>>> *Cc:* pulp-list at redhat.com
>>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>>> pulp on Amazon EC2 (pulp3)
>>>
>>>
>>> PR, as suggested: https://github.com/pulp/pulpcore/pull/366.
>>>
>>>
>>> Cheers,
>>>
>>>
>>> Joey Dumont
>>>
>>> Technical Advisor, Knowledge, Information, and Technology Services
>>> National Research Council Canada / Governement of Canada
>>> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>>
>>> Conseiller technique, Services du savoir, de l'information et de la
>>> technologie
>>> Conseil national de recherches Canada / Gouvernement du Canada
>>> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>>> 438-340-7436
>>> ------------------------------
>>> *From:* Mike DePaulo <mikedep333 at redhat.com>
>>> *Sent:* 06 November 2019 10:52
>>> *To:* Dumont, Joey
>>> *Cc:* David Davis; pulp-list at redhat.com
>>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>>> pulp on Amazon EC2 (pulp3)
>>>
>>> Hi Joey,
>>>
>>> It sounds like aws_default_acl should be documented here then:
>>>
>>> https://docs.pulpproject.org/en/3.0/nightly/installation/storage.html?highlight=aws
>>>
>>> Care to submit a documentation PR?
>>>
>>> https://github.com/pulp/pulpcore/blob/master/docs/installation/storage.rst
>>>
>>> Thanks,
>>> -Mike
>>>
>>> On Wed, Nov 6, 2019 at 9:07 AM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
>>> wrote:
>>>
>>>> Turns out the issue was on my end. I had to add aws_default_acl: None
>>>> to the pulp_settings section of the playbook. The public-read ACL was
>>>> incompatible with the BlockPublicAccess settings that I had set on my S3
>>>> bucket.
>>>>
>>>> I'm now encountering a different issue, but I'll start another thread
>>>> for that one.
>>>>
>>>> Thanks for the pointers, they were very helpful!
>>>>
>>>> Joey Dumont
>>>>
>>>> Technical Advisor, Knowledge, Information, and Technology Services
>>>> National Research Council Canada / Governement of Canada
>>>> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>>>
>>>> Conseiller technique, Services du savoir, de l'information et de la
>>>> technologie
>>>> Conseil national de recherches Canada / Gouvernement du Canada
>>>> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>>>> 438-340-7436
>>>> ------------------------------
>>>> *From:* David Davis <daviddavis at redhat.com>
>>>> *Sent:* 01 November 2019 15:51
>>>> *To:* Dumont, Joey
>>>> *Cc:* pulp-list at redhat.com
>>>> *Subject:* Re: [Pulp-list] Issues with using S3 storage when running
>>>> pulp on Amazon EC2 (pulp3)
>>>>
>>>> Unfortunately I don't know of a good way to debug the problem other
>>>> than to dig into the code. If you want to debug from the Pulp code, you
>>>> could stick a debugger in the artifact saver stage:
>>>>
>>>>
>>>> https://github.com/pulp/pulpcore/blob/2203fee1407738a4ddd8e644fcbc741aab0bca63/pulpcore/plugin/stages/artifact_stages.py#L179-L200
>>>>
>>>> What I would probably do though is stick a debug statement here in
>>>> django-storages to see what params it's passing to boto3:
>>>>
>>>>
>>>> https://github.com/jschneier/django-storages/blob/0ab2b1e3efd2bcaf0f24540a718993acc7742d9b/storages/backends/s3boto3.py#L511
>>>>
>>>> You can see the location of django-storages with `pip show
>>>> django-storages`.
>>>>
>>>> Sorry I don't have a better answer for you. Perhaps this is something
>>>> we can improve in the future. Also, I'd be curious as to what the issue is
>>>> as it sounds like everything should work in theory.
>>>>
>>>> David
>>>>
>>>>
>>>> On Fri, Nov 1, 2019 at 2:26 PM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
>>>> wrote:
>>>>
>>>>> I've installed the latest pulp3 using the Ansible installer using the
>>>>> following playbook:
>>>>>
>>>>>
>>>>> ---
>>>>> - hosts: mirrors
>>>>> vars:
>>>>> prereq_pip_packages:
>>>>> - django-storages
>>>>> - boto3
>>>>> pulp_use_system_wide_pkgs: True
>>>>> pulp_default_admin_password: !vault |
>>>>> $ANSIBLE_VAULT;1.1;AES256
>>>>> ...
>>>>> pulp_settings:
>>>>> secret_key: !vault |
>>>>> $ANSIBLE_VAULT;1.1;AES256
>>>>> ...
>>>>> default_file_storage: 'storages.backends.s3boto3.S3Boto3Storage'
>>>>> aws_storage_bucket_name: 'xxx-pulp-storage'
>>>>> aws_s3_region_name: 'ca-central-1'
>>>>> aws_s3_addressing_style: "path"
>>>>> media_root: '/pulp3/'
>>>>> pulp_install_plugins:
>>>>> pulp-file: {}
>>>>> pulp-rpm:
>>>>> prereq_role: "pulp.pulp_rpm_prerequisites"
>>>>> # pulp-docker: {}
>>>>> roles:
>>>>> - pulp-database
>>>>> - pulp-workers
>>>>> - pulp-resource-manager
>>>>> - pulp-webserver
>>>>> - pulp-content
>>>>> environment:
>>>>> DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>>>>>
>>>>> I also set up an RPM repo that uses S3 for storage. However, when I
>>>>> try to sync, I get an AccessDenied error. I know the instance profile is
>>>>> correct, as I can upload objects from that instance using both the AWS CLI
>>>>> and Boto3 without specifying credentials.
>>>>>
>>>>> How can I debug this further? Is there a way for me know what
>>>>> parameters are passed to the put_object boto3 call by the sync task?
>>>>>
>>>>> Cheers,
>>>>>
>>>>>
>>>>>
>>>>> Joey Dumont
>>>>>
>>>>> Technical Advisor, Knowledge, Information, and Technology Services
>>>>> National Research Council Canada / Governement of Canada
>>>>> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>>>>>
>>>>> Conseiller technique, Services du savoir, de l'information et de la
>>>>> technologie
>>>>> Conseil national de recherches Canada / Gouvernement du Canada
>>>>> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.:
>>>>> 438-340-7436
>>>>> _______________________________________________
>>>>> Pulp-list mailing list
>>>>> Pulp-list at redhat.com
>>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>>
>>>> _______________________________________________
>>>> Pulp-list mailing list
>>>> Pulp-list at redhat.com
>>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>>
>>>
>>>
>>> --
>>>
>>> Mike DePaulo
>>>
>>> He / Him / His
>>>
>>> Service Reliability Engineer, Pulp
>>>
>>> Red Hat <https://www.redhat.com/>
>>>
>>> IM: mikedep333
>>>
>>> GPG: 51745404
>>> <https://www.redhat.com/>
>>> _______________________________________________
>>> Pulp-list mailing list
>>> Pulp-list at redhat.com
>>> https://www.redhat.com/mailman/listinfo/pulp-list
>>
>>
>
> --
>
> Mike DePaulo
>
> He / Him / His
>
> Service Reliability Engineer, Pulp
>
> Red Hat <https://www.redhat.com/>
>
> IM: mikedep333
>
> GPG: 51745404
> <https://www.redhat.com/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20191109/d39fcf0a/attachment.htm>
More information about the Pulp-list
mailing list