[rhos-list] [gluster-swift] Gluster UFO 3.4 swift Multi tenant question

Fri Sep 20 20:11:41 UTC 2013

This is great! Thank you. The only issue is that we use github as a 
public repo and we do not use github for our normal development 
workflow. Do you mind resubmitting the change using the development 
workflow as described 
here:https://github.com/gluster/gluster-swift/blob/master/doc/markdown/dev_guide.md. 
More information can be found at https://launchpad.net/gluster-swift

- Luis

On 09/19/2013 05:23 PM, Paul Robert Marino wrote:
> Thank you every one for your help especially Louis
>
> I tested the RPM and it went well every thing is working now.
>
> I did have to use the tenant ID's as the volume names. I may submit an
> update to the documentation to clarify this for people
>
> So in other words the volume names have to match the output of
>
> "
> keystone tenant-list|grep -v + | \
> grep -v -P '^\|\s+id\s+\|\s+name\s+\|\s+enabled\s+\|$' | \
> grep -v -P '^\w+:' | awk '{print $2}'
> "
>
>
> I've created an updated copy of gluster-swift-gen-builders that grabs
> the value of mount_ip from /etc/swift/fs.conf and posted it on github.
> you should see a pull request on the site for the submission. of the
> change
>
> Thank you every one for your help
>
> On Tue, Sep 17, 2013 at 4:38 PM, Paul Robert Marino <prmarino1 at gmail.com> wrote:
>> Luis
>> Thanks for the timely response.
>>
>> On Tue, Sep 17, 2013 at 1:52 PM, Luis Pabon <lpabon at redhat.com> wrote:
>>> On 09/17/2013 11:13 AM, Paul Robert Marino wrote:
>>>> Luis
>>>> well thats intresting because it was my impression that Gluster UFO
>>>> 3.4 was based on the Grizzly version of Swift.
>>> [LP] Sorry, the gluster-ufo RPM is Essex only.
>> [PRM] The source of my confusion was here
>> http://www.gluster.org/community/documentation/index.php/Features34
>> and here http://www.gluster.org/2013/06/glusterfs-3-4-and-swift-where-are-all-the-pieces/
>> These pages on the gluster site should probably be updated to reflect
>> the changes.
>>
>>
>>>
>>>> Also I was previously unaware of this new rpm which doesnt seem to be
>>>> in a repo any where.
>>> [LP] gluster-swift project RPMs have been submitted to Fedora and are
>>> currently being reviewed.
>> [PRM] Cool if they are in the EPEL testing repo Ill look for them
>> there because I would rather pull the properly EPEL signed RPMs if
>> they exist just to make node deployments easier. If not Ill ask some
>> of my friends offline if they can help expedite it.
>>
>>>
>>>> also there is a line in this new howto that is extreamly unclear
>>>>
>>>> "
>>>> /usr/bin/gluster-swift-gen-builders test
>>>> "
>>>> in place of "test" what should go there is it the tenant ID string,
>>>> the tenant name, or just a generic volume you can name whatever you
>>>> want?
>>>> in other words how should the Gluster volumes be named?
>>> [LP] We will clarify that in the quick start guide.  Thank you for pointing
>>> it out.  While we update the community site, please refer to the
>>> documentation available here http://goo.gl/bQFI8o for a usage guide.
>>>
>>> As for the tool, the format is:
>>> gluster-swift-gen-buildes [VOLUME] [VOLUME...]
>>>
>>> Where VOLUME is the name of the GlusterFS volume to use for object storage.
>>> For example
>>> if the following two GlusterFS volumes, volume1 and volume2, need to be
>>> accessed over Swift,
>>> then you can type the following:
>>>
>>> # gluster-swift-gen-builders volume1 volume2
>> [PRM] That part I understood however it doesn't answer the question exactly.
>>
>> Correct me if I'm wrong but looking over the code briefly it looks as
>> though the volume name needs to be the same as the tenant ID number
>> like it did with Gluster UFO 3.3.
>> so for example
>> if I do a " keystone tenant-list" and a see tenant1 with an id of
>> "f6da0a8151ff43b7be10d961a20c94d6" then I would need to create a
>> volume named f6da0a8151ff43b7be10d961a20c94d6
>>
>> If I can name the volumes whatever I want or give them the same name
>> as the tenant that would be great because it makes it easier for other
>> SA's who are not directly working with OpenStack but may need to mount
>> the volumes to comprehend, but its not urgently needed.
>>
>> One thing I was glad to see is that with Gluster UFO 3.3 I had to add
>> mount points to /etc/fstab for each volume and manually create the
>> directories for the mount points this looks to have been corrected in
>> Gluster-Swift.
>>
>>> For more information please read: http://goo.gl/gd8LkW
>>>
>>> Let us know if you have any more questions or comments.
>> [PRM] I may fork the Github repo and add some changes that may be
>> beneficial so they can be reviewed and possibly merged.
>> for example it would be nice if the  gluster-swift-gen-buildes script
>> used the value of the mount_ip field in /etc/swift/fs.conf instead of
>> 127.0.0.1 if its defined.
>> also I might make a more robust version that allows create, add,
>> remove, and list options.
>>
>>
>> Ill do testing tomorrow and let everyone know how it goes.
>>
>>
>>> - Luis
>>>
>>>>
>>>> On Tue, Sep 17, 2013 at 10:10 AM, Luis Pabon <lpabon at redhat.com> wrote:
>>>>> First thing I can see is that you have Essex based gluster-ufo-* which
>>>>> has
>>>>> been replaced by the gluster-swift project.  We are currently in progress
>>>>> of
>>>>> replacing the gluster-ufo-* with RPMs from the gluster-swift project in
>>>>> Fedora.
>>>>>
>>>>> Please checkout the following quickstart guide which show how to download
>>>>> the Grizzly version of gluster-swift:
>>>>>
>>>>> https://github.com/gluster/gluster-swift/blob/master/doc/markdown/quick_start_guide.md
>>>>> .
>>>>>
>>>>> For more information please visit: https://launchpad.net/gluster-swift
>>>>>
>>>>> - Luis
>>>>>
>>>>>
>>>>> On 09/16/2013 05:02 PM, Paul Robert Marino wrote:
>>>>>
>>>>> Sorry for the delay on reporting the details. I got temporarily pulled
>>>>> off the project and dedicated to a different project which was
>>>>> considered higher priority by my employer. I'm just getting back to
>>>>> doing my normal work today.
>>>>>
>>>>> first here are the rpms I have installed
>>>>> "
>>>>>    rpm -qa |grep -P -i '(gluster|swift)'
>>>>> glusterfs-libs-3.4.0-8.el6.x86_64
>>>>> glusterfs-server-3.4.0-8.el6.x86_64
>>>>> openstack-swift-plugin-swift3-1.0.0-0.20120711git.el6.noarch
>>>>> openstack-swift-proxy-1.8.0-2.el6.noarch
>>>>> glusterfs-3.4.0-8.el6.x86_64
>>>>> glusterfs-cli-3.4.0-8.el6.x86_64
>>>>> glusterfs-geo-replication-3.4.0-8.el6.x86_64
>>>>> glusterfs-api-3.4.0-8.el6.x86_64
>>>>> openstack-swift-1.8.0-2.el6.noarch
>>>>> openstack-swift-container-1.8.0-2.el6.noarch
>>>>> openstack-swift-object-1.8.0-2.el6.noarch
>>>>> glusterfs-fuse-3.4.0-8.el6.x86_64
>>>>> glusterfs-rdma-3.4.0-8.el6.x86_64
>>>>> openstack-swift-account-1.8.0-2.el6.noarch
>>>>> glusterfs-ufo-3.4.0-8.el6.noarch
>>>>> glusterfs-vim-3.2.7-1.el6.x86_64
>>>>> python-swiftclient-1.4.0-1.el6.noarch
>>>>>
>>>>> here are some key config files note I've changed the passwords I'm
>>>>> using and hostnames
>>>>> "
>>>>>    cat /etc/swift/account-server.conf
>>>>> [DEFAULT]
>>>>> mount_check = true
>>>>> bind_port = 6012
>>>>> user = root
>>>>> log_facility = LOG_LOCAL2
>>>>> devices = /swift/tenants/
>>>>>
>>>>> [pipeline:main]
>>>>> pipeline = account-server
>>>>>
>>>>> [app:account-server]
>>>>> use = egg:gluster_swift_ufo#account
>>>>> log_name = account-server
>>>>> log_level = DEBUG
>>>>> log_requests = true
>>>>>
>>>>> [account-replicator]
>>>>> vm_test_mode = yes
>>>>>
>>>>> [account-auditor]
>>>>>
>>>>> [account-reaper]
>>>>>
>>>>> "
>>>>>
>>>>> "
>>>>>    cat /etc/swift/container-server.conf
>>>>> [DEFAULT]
>>>>> devices = /swift/tenants/
>>>>> mount_check = true
>>>>> bind_port = 6011
>>>>> user = root
>>>>> log_facility = LOG_LOCAL2
>>>>>
>>>>> [pipeline:main]
>>>>> pipeline = container-server
>>>>>
>>>>> [app:container-server]
>>>>> use = egg:gluster_swift_ufo#container
>>>>>
>>>>> [container-replicator]
>>>>> vm_test_mode = yes
>>>>>
>>>>> [container-updater]
>>>>>
>>>>> [container-auditor]
>>>>>
>>>>> [container-sync]
>>>>> "
>>>>>
>>>>> "
>>>>>    cat /etc/swift/object-server.conf
>>>>> [DEFAULT]
>>>>> mount_check = true
>>>>> bind_port = 6010
>>>>> user = root
>>>>> log_facility = LOG_LOCAL2
>>>>> devices = /swift/tenants/
>>>>>
>>>>> [pipeline:main]
>>>>> pipeline = object-server
>>>>>
>>>>> [app:object-server]
>>>>> use = egg:gluster_swift_ufo#object
>>>>>
>>>>> [object-replicator]
>>>>> vm_test_mode = yes
>>>>>
>>>>> [object-updater]
>>>>>
>>>>> [object-auditor]
>>>>> "
>>>>>
>>>>> "
>>>>> cat /etc/swift/proxy-server.conf
>>>>> [DEFAULT]
>>>>> bind_port = 8080
>>>>> user = root
>>>>> log_facility = LOG_LOCAL1
>>>>> log_name = swift
>>>>> log_level = DEBUG
>>>>> log_headers = True
>>>>>
>>>>> [pipeline:main]
>>>>> pipeline = healthcheck cache authtoken keystone proxy-server
>>>>>
>>>>> [app:proxy-server]
>>>>> use = egg:gluster_swift_ufo#proxy
>>>>> allow_account_management = true
>>>>> account_autocreate = true
>>>>>
>>>>> [filter:tempauth]
>>>>> use = egg:swift#tempauth
>>>>> # Here you need to add users explicitly. See the OpenStack Swift
>>>>> Deployment
>>>>> # Guide for more information. The user and user64 directives take the
>>>>> # following form:
>>>>> #     user_<account>_<username> = <key> [group] [group] [...]
>>>>> [storage_url]
>>>>> #     user64_<account_b64>_<username_b64> = <key> [group] [group]
>>>>> [...] [storage_url]
>>>>> # Where you use user64 for accounts and/or usernames that include
>>>>> underscores.
>>>>> #
>>>>> # NOTE (and WARNING): The account name must match the device name
>>>>> specified
>>>>> # when generating the account, container, and object build rings.
>>>>> #
>>>>> # E.g.
>>>>> #     user_ufo0_admin = abc123 .admin
>>>>>
>>>>> [filter:healthcheck]
>>>>> use = egg:swift#healthcheck
>>>>>
>>>>> [filter:cache]
>>>>> use = egg:swift#memcache
>>>>>
>>>>>
>>>>> [filter:keystone]
>>>>> use = egg:swift#keystoneauth
>>>>> #paste.filter_factory = keystone.middleware.swift_auth:filter_factory
>>>>> operator_roles = Member,admin,swiftoperator
>>>>>
>>>>>
>>>>> [filter:authtoken]
>>>>> paste.filter_factory = keystone.middleware.auth_token:filter_factory
>>>>> auth_host = keystone01.vip.my.net
>>>>> auth_port = 35357
>>>>> auth_protocol = http
>>>>> admin_user = swift
>>>>> admin_password = PASSWORD
>>>>> admin_tenant_name = service
>>>>> signing_dir = /var/cache/swift
>>>>> service_port = 5000
>>>>> service_host = keystone01.vip.my.net
>>>>>
>>>>> [filter:swiftauth]
>>>>> use = egg:keystone#swiftauth
>>>>> auth_host = keystone01.vip.my.net
>>>>> auth_port = 35357
>>>>> auth_protocol = http
>>>>> keystone_url = https://keystone01.vip.my.net:5000/v2.0
>>>>> admin_user = swift
>>>>> admin_password = PASSWORD
>>>>> admin_tenant_name = service
>>>>> signing_dir = /var/cache/swift
>>>>> keystone_swift_operator_roles = Member,admin,swiftoperator
>>>>> keystone_tenant_user_admin = true
>>>>>
>>>>> [filter:catch_errors]
>>>>> use = egg:swift#catch_errors
>>>>> "
>>>>>
>>>>> "
>>>>> cat /etc/swift/swift.conf
>>>>> [DEFAULT]
>>>>>
>>>>>
>>>>> [swift-hash]
>>>>> # random unique string that can never change (DO NOT LOSE)
>>>>> swift_hash_path_suffix = gluster
>>>>> #3d60c9458bb77abe
>>>>>
>>>>>
>>>>> # The swift-constraints section sets the basic constraints on data
>>>>> # saved in the swift cluster.
>>>>>
>>>>> [swift-constraints]
>>>>>
>>>>> # max_file_size is the largest "normal" object that can be saved in
>>>>> # the cluster. This is also the limit on the size of each segment of
>>>>> # a "large" object when using the large object manifest support.
>>>>> # This value is set in bytes. Setting it to lower than 1MiB will cause
>>>>> # some tests to fail. It is STRONGLY recommended to leave this value at
>>>>> # the default (5 * 2**30 + 2).
>>>>>
>>>>> # FIXME: Really? Gluster can handle a 2^64 sized file? And can the
>>>>> fronting
>>>>> # web service handle such a size? I think with UFO, we need to keep with
>>>>> the
>>>>> # default size from Swift and encourage users to research what size their
>>>>> web
>>>>> # services infrastructure can handle.
>>>>>
>>>>> max_file_size = 18446744073709551616
>>>>>
>>>>>
>>>>> # max_meta_name_length is the max number of bytes in the utf8 encoding
>>>>> # of the name portion of a metadata header.
>>>>>
>>>>> #max_meta_name_length = 128
>>>>>
>>>>>
>>>>> # max_meta_value_length is the max number of bytes in the utf8 encoding
>>>>> # of a metadata value
>>>>>
>>>>> #max_meta_value_length = 256
>>>>>
>>>>>
>>>>> # max_meta_count is the max number of metadata keys that can be stored
>>>>> # on a single account, container, or object
>>>>>
>>>>> #max_meta_count = 90
>>>>>
>>>>>
>>>>> # max_meta_overall_size is the max number of bytes in the utf8 encoding
>>>>> # of the metadata (keys + values)
>>>>>
>>>>> #max_meta_overall_size = 4096
>>>>>
>>>>>
>>>>> # max_object_name_length is the max number of bytes in the utf8 encoding
>>>>> of
>>>>> an
>>>>> # object name: Gluster FS can handle much longer file names, but the
>>>>> length
>>>>> # between the slashes of the URL is handled below. Remember that most web
>>>>> # clients can't handle anything greater than 2048, and those that do are
>>>>> # rather clumsy.
>>>>>
>>>>> max_object_name_length = 2048
>>>>>
>>>>> # max_object_name_component_length (GlusterFS) is the max number of bytes
>>>>> in
>>>>> # the utf8 encoding of an object name component (the part between the
>>>>> # slashes); this is a limit imposed by the underlying file system (for
>>>>> XFS
>>>>> it
>>>>> # is 255 bytes).
>>>>>
>>>>> max_object_name_component_length = 255
>>>>>
>>>>> # container_listing_limit is the default (and max) number of items
>>>>> # returned for a container listing request
>>>>>
>>>>> #container_listing_limit = 10000
>>>>>
>>>>>
>>>>> # account_listing_limit is the default (and max) number of items returned
>>>>> # for an account listing request
>>>>>
>>>>> #account_listing_limit = 10000
>>>>>
>>>>>
>>>>> # max_account_name_length is the max number of bytes in the utf8 encoding
>>>>> of
>>>>> # an account name: Gluster FS Filename limit (XFS limit?), must be the
>>>>> same
>>>>> # size as max_object_name_component_length above.
>>>>>
>>>>> max_account_name_length = 255
>>>>>
>>>>>
>>>>> # max_container_name_length is the max number of bytes in the utf8
>>>>> encoding
>>>>> # of a container name: Gluster FS Filename limit (XFS limit?), must be
>>>>> the
>>>>> same
>>>>> # size as max_object_name_component_length above.
>>>>>
>>>>> max_container_name_length = 255
>>>>>
>>>>> "
>>>>>
>>>>>
>>>>> The volumes
>>>>> "
>>>>>    gluster volume list
>>>>> cindervol
>>>>> unified-storage-vol
>>>>> a07d2f39117c4e5abdeba722cf245828
>>>>> bd74a005f08541b9989e392a689be2fc
>>>>> f6da0a8151ff43b7be10d961a20c94d6
>>>>> "
>>>>>
>>>>> if I run the command
>>>>> "
>>>>>    gluster-swift-gen-builders unified-storage-vol
>>>>> a07d2f39117c4e5abdeba722cf245828 bd74a005f08541b9989e392a689be2fc
>>>>> f6da0a8151ff43b7be10d961a20c94d6
>>>>> "
>>>>>
>>>>> because of a change in the script in this version as compaired to the
>>>>> version I got from
>>>>> http://repos.fedorapeople.org/repos/kkeithle/glusterfs/ the
>>>>> gluster-swift-gen-builders script only takes the first option and
>>>>> ignores the rest.
>>>>>
>>>>> other than the location of the config files none of the changes Ive
>>>>> made are functionally different than the ones mentioned in
>>>>>
>>>>> http://www.gluster.org/2012/09/howto-using-ufo-swift-a-quick-and-dirty-setup-guide/
>>>>>
>>>>> The result is that the first volume named "unified-storage-vol" winds
>>>>> up being used for every thing regardless of the tenant, and users and
>>>>> see and manage each others objects regardless of what tenant they are
>>>>> members of.
>>>>> through the swift command or via horizon.
>>>>>
>>>>> In a way this is a good thing for me it simplifies thing significantly
>>>>> and would be fine if it just created a directory for each tenant and
>>>>> only allow the user to access the individual directories, not the
>>>>> whole gluster volume.
>>>>> by the way seeing every thing includes the service tenants data so
>>>>> unprivileged users can delete glance images without being a member of
>>>>> the service group.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Sep 2, 2013 at 9:58 PM, Paul Robert Marino <prmarino1 at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Well I'll give you the full details in the morning but simply I used the
>>>>> stock cluster ring builder script that came with the 3.4 rpms and the old
>>>>> version from 3.3 took the list of volumes and would add all of them the
>>>>> version with 3.4 only takes the first one.
>>>>>
>>>>> Well I ran the script expecting the same behavior but instead they all
>>>>> used
>>>>> the first volume in the list.
>>>>>
>>>>> Now I knew from the docs I read that the per tenant directories in a
>>>>> single
>>>>> volume were one possible plan for 3.4 to deal with the scalding issue
>>>>> with a
>>>>> large number of tenants, so when I saw the difference in the script and
>>>>> that
>>>>> it worked I just assumed that this was done and I missed something.
>>>>>
>>>>>
>>>>>
>>>>> -- Sent from my HP Pre3
>>>>>
>>>>> ________________________________
>>>>> On Sep 2, 2013 20:55, Ramana Raja <rraja at redhat.com> wrote:
>>>>>
>>>>> Hi Paul,
>>>>>
>>>>> Currently, gluster-swift doesn't support the feature of multiple
>>>>> accounts/tenants accessing the same volume. Each tenant still needs his
>>>>> own
>>>>> gluster volume. So I'm wondering how you were able to observe the
>>>>> reported
>>>>> behaviour.
>>>>>
>>>>> How did you prepare the ringfiles for the different tenants, which use
>>>>> the
>>>>> same gluster volume? Did you change the configuration of the servers?
>>>>> Also,
>>>>> how did you access the files that you mention? It'd be helpful if you
>>>>> could
>>>>> share the commands you used to perform these actions.
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Ram
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Vijay Bellur" <vbellur at redhat.com>
>>>>> To: "Paul Robert Marino" <prmarino1 at gmail.com>
>>>>> Cc: rhos-list at redhat.com, "Luis Pabon" <lpabon at redhat.com>, "Ramana Raja"
>>>>> <rraja at redhat.com>, "Chetan Risbud" <crisbud at redhat.com>
>>>>> Sent: Monday, September 2, 2013 4:17:51 PM
>>>>> Subject: Re: [rhos-list] Gluster UFO 3.4 swift Multi tenant question
>>>>>
>>>>> On 09/02/2013 01:39 AM, Paul Robert Marino wrote:
>>>>>
>>>>> I have Gluster UFO installed as a back end for swift from here
>>>>> http://download.gluster.org/pub/gluster/glusterfs/3.4/3.4.0/RHEL/epel-6/
>>>>> with RDO 3
>>>>>
>>>>> Its working well except for one thing. All of the tenants are seeing
>>>>> one Gluster volume which is some what nice, especially when compared
>>>>> to the old 3.3 behavior of creating one volume per tenant named after
>>>>> the tenant ID number.
>>>>>
>>>>> The problem is I expected to see is sub directory created under the
>>>>> volume root for each tenant but instead what in seeing is that all of
>>>>> the tenants can see the root of the Gluster volume. The result is that
>>>>> all of the tenants can access each others files and even delete them.
>>>>> even scarier is that the tennants can see and delete each others
>>>>> glance images and snapshots.
>>>>>
>>>>> Can any one suggest options to look at or documents to read to try to
>>>>> figure out how to modify the behavior?
>>>>>
>>>>> Adding gluster swift developers who might be able to help.
>>>>>
>>>>> -Vijay
>>>>>
>>>>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/rhos-list/attachments/20130920/2d877beb/attachment.htm>