[rhos-list] [gluster-swift] Gluster UFO 3.4 swift Multi tenant question

Tue Sep 17 17:52:12 UTC 2013

On 09/17/2013 11:13 AM, Paul Robert Marino wrote:
> Luis
> well thats intresting because it was my impression that Gluster UFO
> 3.4 was based on the Grizzly version of Swift.
[LP] Sorry, the gluster-ufo RPM is Essex only.

> Also I was previously unaware of this new rpm which doesnt seem to be
> in a repo any where.
[LP] gluster-swift project RPMs have been submitted to Fedora and are 
currently being reviewed.

> also there is a line in this new howto that is extreamly unclear
>
> "
> /usr/bin/gluster-swift-gen-builders test
> "
> in place of "test" what should go there is it the tenant ID string,
> the tenant name, or just a generic volume you can name whatever you
> want?
> in other words how should the Gluster volumes be named?
[LP] We will clarify that in the quick start guide.  Thank you for 
pointing it out.  While we update the community site, please refer to 
the documentation available here http://goo.gl/bQFI8o for a usage guide.

As for the tool, the format is:
gluster-swift-gen-buildes [VOLUME] [VOLUME...]

Where VOLUME is the name of the GlusterFS volume to use for object 
storage. For example
if the following two GlusterFS volumes, volume1 and volume2, need to be 
accessed over Swift,
then you can type the following:

# gluster-swift-gen-builders volume1 volume2

For more information please read: http://goo.gl/gd8LkW

Let us know if you have any more questions or comments.

- Luis
>
>
> On Tue, Sep 17, 2013 at 10:10 AM, Luis Pabon <lpabon at redhat.com> wrote:
>> First thing I can see is that you have Essex based gluster-ufo-* which has
>> been replaced by the gluster-swift project.  We are currently in progress of
>> replacing the gluster-ufo-* with RPMs from the gluster-swift project in
>> Fedora.
>>
>> Please checkout the following quickstart guide which show how to download
>> the Grizzly version of gluster-swift:
>> https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md
>> .
>>
>> For more information please visit: https://launchpad.net/gluster-swift
>>
>> - Luis
>>
>>
>> On 09/16/2013 05:02 PM, Paul Robert Marino wrote:
>>
>> Sorry for the delay on reporting the details. I got temporarily pulled
>> off the project and dedicated to a different project which was
>> considered higher priority by my employer. I'm just getting back to
>> doing my normal work today.
>>
>> first here are the rpms I have installed
>> "
>>   rpm -qa |grep -P -i '(gluster|swift)'
>> glusterfs-libs-3.4.0-8.el6.x86_64
>> glusterfs-server-3.4.0-8.el6.x86_64
>> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
>> openstack-swift-proxy-1.8.0-2.el6.noarch
>> glusterfs-3.4.0-8.el6.x86_64
>> glusterfs-cli-3.4.0-8.el6.x86_64
>> glusterfs-geo-replication-3.4.0-8.el6.x86_64
>> glusterfs-api-3.4.0-8.el6.x86_64
>> openstack-swift-1.8.0-2.el6.noarch
>> openstack-swift-container-1.8.0-2.el6.noarch
>> openstack-swift-object-1.8.0-2.el6.noarch
>> glusterfs-fuse-3.4.0-8.el6.x86_64
>> glusterfs-rdma-3.4.0-8.el6.x86_64
>> openstack-swift-account-1.8.0-2.el6.noarch
>> glusterfs-ufo-3.4.0-8.el6.noarch
>> glusterfs-vim-3.2.7-1.el6.x86_64
>> python-swiftclient-1.4.0-1.el6.noarch
>>
>> here are some key config files note I've changed the passwords I'm
>> using and hostnames
>> "
>>   cat /etc/swift/account-server.conf
>> [DEFAULT]
>> mount_check = true
>> bind_port = 6012
>> user = root
>> log_facility = LOG_LOCAL2
>> devices = /swift/tenants/
>>
>> [pipeline:main]
>> pipeline = account-server
>>
>> [app:account-server]
>> use = egg:gluster_swift_ufo#account
>> log_name = account-server
>> log_level = DEBUG
>> log_requests = true
>>
>> [account-replicator]
>> vm_test_mode = yes
>>
>> [account-auditor]
>>
>> [account-reaper]
>>
>> "
>>
>> "
>>   cat /etc/swift/container-server.conf
>> [DEFAULT]
>> devices = /swift/tenants/
>> mount_check = true
>> bind_port = 6011
>> user = root
>> log_facility = LOG_LOCAL2
>>
>> [pipeline:main]
>> pipeline = container-server
>>
>> [app:container-server]
>> use = egg:gluster_swift_ufo#container
>>
>> [container-replicator]
>> vm_test_mode = yes
>>
>> [container-updater]
>>
>> [container-auditor]
>>
>> [container-sync]
>> "
>>
>> "
>>   cat /etc/swift/object-server.conf
>> [DEFAULT]
>> mount_check = true
>> bind_port = 6010
>> user = root
>> log_facility = LOG_LOCAL2
>> devices = /swift/tenants/
>>
>> [pipeline:main]
>> pipeline = object-server
>>
>> [app:object-server]
>> use = egg:gluster_swift_ufo#object
>>
>> [object-replicator]
>> vm_test_mode = yes
>>
>> [object-updater]
>>
>> [object-auditor]
>> "
>>
>> "
>> cat /etc/swift/proxy-server.conf
>> [DEFAULT]
>> bind_port = 8080
>> user = root
>> log_facility = LOG_LOCAL1
>> log_name = swift
>> log_level = DEBUG
>> log_headers = True
>>
>> [pipeline:main]
>> pipeline = healthcheck cache authtoken keystone proxy-server
>>
>> [app:proxy-server]
>> use = egg:gluster_swift_ufo#proxy
>> allow_account_management = true
>> account_autocreate = true
>>
>> [filter:tempauth]
>> use = egg:swift#tempauth
>> # Here you need to add users explicitly. See the OpenStack Swift Deployment
>> # Guide for more information. The user and user64 directives take the
>> # following form:
>> #     user_<account>_<username> = <key> [group] [group] [...] [storage_url]
>> #     user64_<account_b64>_<username_b64> = <key> [group] [group]
>> [...] [storage_url]
>> # Where you use user64 for accounts and/or usernames that include
>> underscores.
>> #
>> # NOTE (and WARNING): The account name must match the device name specified
>> # when generating the account, container, and object build rings.
>> #
>> # E.g.
>> #     user_ufo0_admin = abc123 .admin
>>
>> [filter:healthcheck]
>> use = egg:swift#healthcheck
>>
>> [filter:cache]
>> use = egg:swift#memcache
>>
>>
>> [filter:keystone]
>> use = egg:swift#keystoneauth
>> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory
>> operator_roles = Member,admin,swiftoperator
>>
>>
>> [filter:authtoken]
>> paste.filter_factory = keystone.middleware.auth_token:filter_factory
>> auth_host = keystone01.vip.my.net
>> auth_port = 35357
>> auth_protocol = http
>> admin_user = swift
>> admin_password = PASSWORD
>> admin_tenant_name = service
>> signing_dir = /var/cache/swift
>> service_port = 5000
>> service_host = keystone01.vip.my.net
>>
>> [filter:swiftauth]
>> use = egg:keystone#swiftauth
>> auth_host = keystone01.vip.my.net
>> auth_port = 35357
>> auth_protocol = http
>> keystone_url = https://keystone01.vip.my.net:5000/v2.0
>> admin_user = swift
>> admin_password = PASSWORD
>> admin_tenant_name = service
>> signing_dir = /var/cache/swift
>> keystone_swift_operator_roles = Member,admin,swiftoperator
>> keystone_tenant_user_admin = true
>>
>> [filter:catch_errors]
>> use = egg:swift#catch_errors
>> "
>>
>> "
>> cat /etc/swift/swift.conf
>> [DEFAULT]
>>
>>
>> [swift-hash]
>> # random unique string that can never change (DO NOT LOSE)
>> swift_hash_path_suffix = gluster
>> #3d60c9458bb77abe
>>
>>
>> # The swift-constraints section sets the basic constraints on data
>> # saved in the swift cluster.
>>
>> [swift-constraints]
>>
>> # max_file_size is the largest "normal" object that can be saved in
>> # the cluster. This is also the limit on the size of each segment of
>> # a "large" object when using the large object manifest support.
>> # This value is set in bytes. Setting it to lower than 1MiB will cause
>> # some tests to fail. It is STRONGLY recommended to leave this value at
>> # the default (5 * 2**30 + 2).
>>
>> # FIXME: Really? Gluster can handle a 2^64 sized file? And can the fronting
>> # web service handle such a size? I think with UFO, we need to keep with the
>> # default size from Swift and encourage users to research what size their
>> web
>> # services infrastructure can handle.
>>
>> max_file_size = 18446744073709551616
>>
>>
>> # max_meta_name_length is the max number of bytes in the utf8 encoding
>> # of the name portion of a metadata header.
>>
>> #max_meta_name_length = 128
>>
>>
>> # max_meta_value_length is the max number of bytes in the utf8 encoding
>> # of a metadata value
>>
>> #max_meta_value_length = 256
>>
>>
>> # max_meta_count is the max number of metadata keys that can be stored
>> # on a single account, container, or object
>>
>> #max_meta_count = 90
>>
>>
>> # max_meta_overall_size is the max number of bytes in the utf8 encoding
>> # of the metadata (keys + values)
>>
>> #max_meta_overall_size = 4096
>>
>>
>> # max_object_name_length is the max number of bytes in the utf8 encoding of
>> an
>> # object name: Gluster FS can handle much longer file names, but the length
>> # between the slashes of the URL is handled below. Remember that most web
>> # clients can't handle anything greater than 2048, and those that do are
>> # rather clumsy.
>>
>> max_object_name_length = 2048
>>
>> # max_object_name_component_length (GlusterFS) is the max number of bytes in
>> # the utf8 encoding of an object name component (the part between the
>> # slashes); this is a limit imposed by the underlying file system (for XFS
>> it
>> # is 255 bytes).
>>
>> max_object_name_component_length = 255
>>
>> # container_listing_limit is the default (and max) number of items
>> # returned for a container listing request
>>
>> #container_listing_limit = 10000
>>
>>
>> # account_listing_limit is the default (and max) number of items returned
>> # for an account listing request
>>
>> #account_listing_limit = 10000
>>
>>
>> # max_account_name_length is the max number of bytes in the utf8 encoding of
>> # an account name: Gluster FS Filename limit (XFS limit?), must be the same
>> # size as max_object_name_component_length above.
>>
>> max_account_name_length = 255
>>
>>
>> # max_container_name_length is the max number of bytes in the utf8 encoding
>> # of a container name: Gluster FS Filename limit (XFS limit?), must be the
>> same
>> # size as max_object_name_component_length above.
>>
>> max_container_name_length = 255
>>
>> "
>>
>>
>> The volumes
>> "
>>   gluster volume list
>> cindervol
>> unified-storage-vol
>> a07d2f39117c4e5abdeba722cf245828
>> bd74a005f08541b9989e392a689be2fc
>> f6da0a8151ff43b7be10d961a20c94d6
>> "
>>
>> if I run the command
>> "
>>   gluster-swift-gen-builders unified-storage-vol
>> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc
>> f6da0a8151ff43b7be10d961a20c94d6
>> "
>>
>> because of a change in the script in this version as compaired to the
>> version I got from
>> http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ the
>> gluster-swift-gen-builders script only takes the first option and
>> ignores the rest.
>>
>> other than the location of the config files none of the changes Ive
>> made are functionally different than the ones mentioned in
>> http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/
>>
>> The result is that the first volume named "unified-storage-vol" winds
>> up being used for every thing regardless of the tenant, and users and
>> see and manage each others objects regardless of what tenant they are
>> members of.
>> through the swift command or via horizon.
>>
>> In a way this is a good thing for me it simplifies thing significantly
>> and would be fine if it just created a directory for each tenant and
>> only allow the user to access the individual directories, not the
>> whole gluster volume.
>> by the way seeing every thing includes the service tenants data so
>> unprivileged users can delete glance images without being a member of
>> the service group.
>>
>>
>>
>>
>> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1 at gmail.com>
>> wrote:
>>
>> Well I'll give you the full details in the morning but simply I used the
>> stock cluster ring builder script that came with the 3.4 rpms and the old
>> version from 3.3 took the list of volumes and would add all of them the
>> version with 3.4 only takes the first one.
>>
>> Well I ran the script expecting the same behavior but instead they all used
>> the first volume in the list.
>>
>> Now I knew from the docs I read that the per tenant directories in a single
>> volume were one possible plan for 3.4 to deal with the scalding issue with a
>> large number of tenants, so when I saw the difference in the script and that
>> it worked I just assumed that this was done and I missed something.
>>
>>
>>
>> -- Sent from my HP Pre3
>>
>> ________________________________
>> On Sep 2, 2013 20:55, Ramana Raja <rraja at redhat.com> wrote:
>>
>> Hi Paul,
>>
>> Currently, gluster-swift doesn't support the feature of multiple
>> accounts/tenants accessing the same volume. Each tenant still needs his own
>> gluster volume. So I'm wondering how you were able to observe the reported
>> behaviour.
>>
>> How did you prepare the ringfiles for the different tenants, which use the
>> same gluster volume? Did you change the configuration of the servers? Also,
>> how did you access the files that you mention? It'd be helpful if you could
>> share the commands you used to perform these actions.
>>
>> Thanks,
>>
>> Ram
>>
>>
>> ----- Original Message -----
>> From: "Vijay Bellur" <vbellur at redhat.com>
>> To: "Paul Robert Marino" <prmarino1 at gmail.com>
>> Cc: rhos-list at redhat.com, "Luis Pabon" <lpabon at redhat.com>, "Ramana Raja"
>> <rraja at redhat.com>, "Chetan Risbud" <crisbud at redhat.com>
>> Sent: Monday, September 2, 2013 4:17:51 PM
>> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question
>>
>> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:
>>
>> I have Gluster UFO installed as a back end for swift from here
>> http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/
>> with RDO 3
>>
>> Its working well except for one thing. All of the tenants are seeing
>> one Gluster volume which is some what nice, especially when compared
>> to the old 3.3 behavior of creating one volume per tenant named after
>> the tenant ID number.
>>
>> The problem is I expected to see is sub directory created under the
>> volume root for each tenant but instead what in seeing is that all of
>> the tenants can see the root of the Gluster volume. The result is that
>> all of the tenants can access each others files and even delete them.
>> even scarier is that the tennants can see and delete each others
>> glance images and snapshots.
>>
>> Can any one suggest options to look at or documents to read to try to
>> figure out how to modify the behavior?
>>
>> Adding gluster swift developers who might be able to help.
>>
>> -Vijay
>>
>>