[Pulp-list] Issues with using S3 storage when running pulp on Amazon EC2 (pulp3)

[David Davis]

Unfortunately I don't know of a good way to debug the problem other than to
dig into the code. If you want to debug from the Pulp code, you could stick
a debugger in the artifact saver stage:

https://github.com/pulp/pulpcore/blob/2203fee1407738a4ddd8e644fcbc741aab0bca63/pulpcore/plugin/stages/artifact_stages.py#L179-L200

What I would probably do though is stick a debug statement here in
django-storages to see what params it's passing to boto3:

https://github.com/jschneier/django-storages/blob/0ab2b1e3efd2bcaf0f24540a718993acc7742d9b/storages/backends/s3boto3.py#L511

You can see the location of django-storages with `pip show django-storages`.

Sorry I don't have a better answer for you. Perhaps this is something we
can improve in the future. Also, I'd be curious as to what the issue is as
it sounds like everything should work in theory.

David


On Fri, Nov 1, 2019 at 2:26 PM Dumont, Joey <Joey.Dumont at nrc-cnrc.gc.ca>
wrote:

> I've installed the latest pulp3 using the Ansible installer using the
> following playbook:
>
>
> ---
> - hosts: mirrors
>   vars:
>     prereq_pip_packages:
>       - django-storages
>       - boto3
>     pulp_use_system_wide_pkgs: True
>     pulp_default_admin_password: !vault |
>           $ANSIBLE_VAULT;1.1;AES256
>           ...
>     pulp_settings:
>       secret_key: !vault |
>           $ANSIBLE_VAULT;1.1;AES256
>           ...
>       default_file_storage: 'storages.backends.s3boto3.S3Boto3Storage'
>       aws_storage_bucket_name: 'xxx-pulp-storage'
>       aws_s3_region_name: 'ca-central-1'
>       aws_s3_addressing_style: "path"
>       media_root: '/pulp3/'
>     pulp_install_plugins:
>       pulp-file: {}
>       pulp-rpm:
>         prereq_role: "pulp.pulp_rpm_prerequisites"
>         #      pulp-docker: {}
>   roles:
>     - pulp-database
>     - pulp-workers
>     - pulp-resource-manager
>     - pulp-webserver
>     - pulp-content
>   environment:
>     DJANGO_SETTINGS_MODULE: pulpcore.app.settings
>
> I also set up an RPM repo that uses S3 for storage. However, when I try to
> sync, I get an AccessDenied error. I know the instance profile is correct,
> as I can upload objects from that instance using both the AWS CLI and Boto3
> without specifying credentials.
>
> How can I debug this further? Is there a way for me know what parameters
> are passed to the put_object boto3 call by the sync task?
>
> Cheers,
>
>
>
> Joey Dumont
>
> Technical Advisor, Knowledge, Information, and Technology Services
> National Research Council Canada / Governement of Canada
> joey.dumont at nrc-cnrc.gc.ca / Tel: 613-990-8152 / Cell: 438-340-7436
>
> Conseiller technique, Services du savoir, de l'information et de la
> technologie
> Conseil national de recherches Canada / Gouvernement du Canada
> joey.dumont at nrc-cnrc.gc.ca / Tél.: 613-990-8152 / Tél. cell.: 438-340-7436
> _______________________________________________
> Pulp-list mailing list
> Pulp-list at redhat.com
> https://www.redhat.com/mailman/listinfo/pulp-list
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/pulp-list/attachments/20191101/0ba84878/attachment.htm>