Hypertext Transfer Protocol Secure (HTTPS) is the secure version of HTTP, uses encrypted communication between the user and the server. HTTPS avoids Man-in-the-Middle-Attack attacks by relying on Secure Socket Layer (SSL) and Transport Layer Security (TLS) protocols to establish an encrypted connection to shuttle data securely between a client and a server.
This blog post takes you step by step through the process of adding SSL/TLS security to Ceph object storage endpoints. Ceph is a scalable, open-source object storage solution that provides Amazon S3 and SWIFT compatible APIs you can use to build your own public or private cloud object storage solution. Learn more about Ceph at Red Hat Ceph Storage.
Setting up domain record sets
Let's begin by setting up a domain name and configuring its record sets. If you already have a domain name you'd like to use for the Ceph endpoint, great. If not, you can purchase one from any domain registrar. In this example, we'll use the domain name ceph-s3.com
.
First, we need to add record sets to the domain such that the domain name can resolve into the IP address of the Ceph RADOS Gateway or the load balancer (LB) that is fronting your Ceph RADOS Gateways. To do this, you need to log in to your domain registrar dashboard and create two record sets:
- A type recordset with Domain Name and IPv4 Address of your Internet-accessible LB or Ceph RGW host. (If you're using IPv6, choose AAAA type.)
- CNAME type recordset Wildcard Subdomain Name and IPv4 Address of your Internet-accessible LB or Ceph RGW host
Note: The reason we chose the wildcard subdomain name is important. We want to resolve all subdomains (e.g., bucket.ceph-s3.com
) to the same IP address, because S3 treats subdomain prefixes as the bucket name.
Domain record set additions are highlighted in the following screenshot:
DNS changes usually take a few tens of minutes to propagate, once the changes are synced. We should be able to ping domain name as well as any subdomain from anywhere on the Internet, provided ICMP Ingress traffic is allowed on the host.
karasing-OSX:~$ ping ceph-s3.com PING ceph-s3.com (54.209.4.196): 56 data bytes 64 bytes from 54.209.4.196: icmp_seq=0 ttl=40 time=1095.231 ms 64 bytes from 54.209.4.196: icmp_seq=1 ttl=40 time=1099.570 ms 64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1199.266 ms 64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1199.266 ms ^C --- ceph-s3.com ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1095.231/1131.356/1199.266/48.053 ms karasing-OSX:~$
karasing-OSX:~$ ping anynameintheuniverse.ceph-s3.com PING 54.209.4.196 (54.209.4.196): 56 data bytes 64 bytes from 54.209.4.196: icmp_seq=0 ttl=40 time=1491.105 ms 64 bytes from 54.209.4.196: icmp_seq=1 ttl=40 time=1262.021 ms 64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1205.943 ms 64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1205.943 ms ^C --- 54.209.4.196 ping statistics --- 4 packets transmitted, 4 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 1205.943/1319.690/1491.105/123.352 ms karasing-OSX:~$
SSL certificate: Installation and setup
A SSL certificate is a set of encrypted files that binds an organization's identity, domain name, IP address, and cryptographic keys. Once these SSL certificates are installed on the host server, a secure connection is allowed between the host server and the client machine. In a client’s Internet browser, a green padlock will appear next to the URL as a visual cue to users that traffic is protected.
Some organizations might desire a more advanced certificate that requires additional validation. These SSL certificates must be purchased from a trusted Certificate Authority (CA). For the sake of demonstration in this example, we'll use Let's Encrypt which is a certificate authority that provides free X.509 certificates for Transport Layer Security (TLS) encryption. [ credits: [wikipedia](https://en.wikipedia.org/wiki/Let's_Encrypt) & thanks Let's Encrypt for your free service]
Next, we'll install epel-release
and certbot,
a CLI tool for requesting SSL certificates, from Let's Encrypt CA.
yum install -y http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm yum install -y certbot
Request for SSL certificate using certbot
CLI client
certbot certonly --manual -d ceph-s3.com -d *.ceph-s3.com --agree-tos manual-public-ip-logging-ok --preferred-challenges dns-01 --server https://acme-v02.api.letsencrypt.org/directory
Note: The first -d option in certbot CLI represent base domain, the subsequent -d options represents sub-domains.
Important: Make sure you are using wildcard (*) for subdomain, because we are requesting a wildcard subdomain SSL certification from Let’s Encrypt. If a certificate is only issued for base domain, it will not be compatible with subdomain prefix notation.
The following snippet shows the output of certbot CLI command. The DNS challenge method will generate two DNS TXT records, which must be added as a TXT record set for your domain (from the domain registrar dashboard):
[root@ceph-admin ~]# certbot certonly --manual -d ceph-s3.com -d *.ceph-s3.com --agree-tos --manual-public-ip-logging-ok --preferred-challenges dns-01 --server https://acme-v02.api.letsencrypt.org/directory Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator manual, Installer None Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org Obtaining a new certificate Performing the following challenges: dns-01 challenge for ceph-s3.com dns-01 challenge for ceph-s3.com ------------------------------------------------------------------------------- Please deploy a DNS TXT record under the name _acme-challenge.ceph-s3.com with the following value: BzL-LXXkDWwdde8RFUnbQ3fdYt5N6ZXELu4T26KIXa4 <== This Value Before continuing, verify the record is deployed. ------------------------------------------------------------------------------- Press Enter to continue ------------------------------------------------------------------------------- Please deploy a DNS TXT record under the name _acme-challenge.ceph-s3.com with the following value: O-_g-eeu4cSI0xXSdrw3OBrWVgzZXJC59Xjkhyk39MQ <== This Value Before continuing, verify the record is deployed.
-------------------------------------------------------------------------------
Keep the certbot CLI command running, and open your domain registrar dashboard. Deploy a new DNS TXT record under the name _acme-challenge.yourdomain.com, and enter both these values (marked with <== in the preceding output) in two separate lines inside double quotes. The following screenshot shows how:
Keep the certbot CLI command running and, once you've successfully added DNS TXT record in your domain record set, open another terminal. Now, we'll verify if DNS TXT records are applied on your domain by running the command host -t txt _acme-challenge.yourdomain.com. You should be able to see the same DNS TXT as mentioned in certbot CLI. If you do not, wait for a few minutes for DNS synchronization to occur.
karasing-OSX:~$ host -t txt _acme-challenge.ceph-s3.com _acme-challenge.ceph-s3.com descriptive text "BzL-LXXkDWwdde8RFUnbQ3fdYt5N6ZXELu4T26KIXa4" _acme-challenge.ceph-s3.com descriptive text "O-_g-eeu4cSI0xXSdrw3OBrWVgzZXJC59Xjkhyk39MQ" karasing-OSX:~$
Once you've verified DNS TXT records are applied to your domain, return to certbot CLI and press Enter to continue. You will be notified that the system is waiting for verification and cleaning up challenges. You should then see the following message:
Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/ceph-s3.com/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/ceph-s3.com/privkey.pem Your cert will expire on 2018-10-07. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run "certbot renew" If you like Certbot, please consider supporting our work by: Donating to ISRG / Let's Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le [root@ceph-admin ~]#
At this point, SSL certificates have been issued to your domain/host, so now we must verify them.
[root@ceph-admin ~]# ls -l /etc/letsencrypt/live/ceph-s3.com/ total 4 lrwxrwxrwx 1 root root 35 Jul 9 19:06 cert.pem -> ../../archive/ceph-s3.com/cert1.pem lrwxrwxrwx 1 root root 36 Jul 9 19:06 chain.pem -> ../../archive/ceph-s3.com/chain1.pem lrwxrwxrwx 1 root root 40 Jul 9 19:06 fullchain.pem -> ../../archive/ceph-s3.com/fullchain1.pem lrwxrwxrwx 1 root root 38 Jul 9 19:06 privkey.pem -> ../../archive/ceph-s3.com/privkey1.pem -rw-r--r-- 1 root root 682 Jul 9 19:06 README [root@ceph-admin ~]#
Installing LB for object storage service
Many like HAProxy, because it's easy and does its job well. In this example, we will use HAProxy to perform SSL termination for our domain name (Ceph object storage endpoint).
Note: Starting with Red Hat Ceph Storage 2, Ceph RADOS Gateway natively supports TLS by relying on OpenSSL Library. You can get more information on native SSL/TLS configuration here. In this example, we specifically choose to terminate SSL at HAProxy level. This gives us an advantage like when we have multiple instances of Ceph RGW we do not need to get multiple domain names/SSL certificates for each of them. One domain name with SSL termination at LB does the job.
Next, we'll install HAproxy on the same host whose public IP has been bound with your domain name, in this case, ceph-s3.com.
yum install -y haproxy
Next, we'll create a certs directory.
mkdir -p /etc/haproxy/certs
Combine certificate files fullchain.pem and privkey.pem into a single file for our domain next.
DOMAIN='ceph-s3.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem
/etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'
The next step is to change the permission of the certs directory.
chmod -R go-rwx /etc/haproxy/certs
Optionally, you can move the haproxy.cfg original file and create a new config file with the following configuration settings:
mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig vim /etc/haproxy/haproxy.cfg
global log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon tune.ssl.default-dh-param 2048 stats socket /var/lib/haproxy/stats defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch option httpchk HEAD / retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 frontend www-http bind 10.100.0.10:80 reqidel ^XForwardedFor:.* reqadd X-Forwarded-Proto:\ http default_backend www-backend option forwardfor frontend www-https bind 10.100.0.10:443 ssl crt /etc/haproxy/certs/ceph-s3.com.pem reqadd X-Forwarded-Proto:\ https acl letsencrypt-acl path_beg /.well-known/acme-challenge/ use_backend letsencrypt-backend if letsencrypt-acl default_backend www-backend backend www-backend redirect scheme https if !{ ssl_fc } server ceph-admin 10.100.0.10:8081 check inter 2000 rise 2 fall 5 backend letsencrypt-backend server letsencrypt 127.0.0.1:54321
As you may have noted, we're using a couple of non-default parameters in the haproxy config file, such as:
- tune.ssl.default-dh-param is required to provide OpenSSL the necessary parameters for the SSL/TLS handshake.
- frontend www-http binds haproxy to port 80 of the local machine and redirects traffic to the default backend www-backend. If a user uses HTTP protocol in the request, it should redirect to HTTPS.
- frontend www-https binds haproxy to port 443 of the local machine. It also redirects traffic to the default backend www-backend and uses the SSL certificate path to encrypt/terminate the traffic.
- frontend www-https also uses letsencrypt-backend if you want to auto-renew the SSL certificate from Let's Encrypt CA.
- backend www-backend simply redirects all the SSL terminated traffic to Ceph RGW node 10.100.0.10:8081. For HA and performance, you must have multiple Ceph RGW instances whose IPs should be added in the same backend section so that HAProxy can load balance among Ceph RGW instances.
Finally, we restart HAproxy and verify its listening on ports 80 and 443:
systemctl start haproxy systemctl status haproxy ; netstat -plunt | grep -i haproxy
Configuring Ceph RGW
Up to this point, we've configured the domain name, set required record sets, generated SSL certificate, configured HAProxy to encrypt/terminate SSL, and redirected traffic to the Ceph RGW instance. Now we need to configure Ceph RGW to listen on port 8081 as configured in HAproxy. To do so, in the Ceph RGW node edit /etc/ceph/ceph.conf and update the client.rgw section as shown in the following:
[client.rgw.ceph-admin] host = ceph-admin keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-admin/keyring log file = /var/log/ceph/ceph-rgw-ceph-admin.log rgw frontends = civetweb port=10.100.0.10:8081 num_threads=512 rgw resolve cname = true rgw dns name = ceph-s3.com
Note: rgw resolve cname = true forces rgw to use the DNS CNAME record of the request hostname field (if the hostname is not equal to rgw dns name).
Note: rgw dns name = ceph-s3.com is the DNS name of the served domain.
Now, we'll restart the Ceph RGW instance and verify its listening on port 8081.
systemctl restart ceph-radosgw@rgw.ceph-admin.service netstat -plunt | grep -i rados
Accessing Ceph object storage secure endpoint
To test the HTTPS-enabled Ceph object storage URL, execute the following curl command or type https://ceph-s3.com in any web browser:
curl https://ceph-s3.com
It should yield output like the following:
[student@ceph-admin ~]$ curl https://ceph-s3.com anonymous [student@ceph-admin ~]$ [student@ceph-admin ~]$
Let's try accessing Ceph object storage using S3cmd:
yum install -y s3cmd
Configure S3cmd CLI by providing config options like access/secret keys, Ceph S3 secure endpoints in host/host-bucket parameters.
s3cmd --access_key=S3user1 --secret_key=S3user1key --host=ceph-s3.com
--host-bucket="%(bucket)s.ceph-s3.com" --dump-config > /home/student/.s3cfg
Note: By default, s3cmd uses HTTPS connection, so there is no need to explicitly specify that.
Next, we'll interact with Ceph object storage using s3cmd ls, s3cmd mb commands:
[student@ceph-admin ~]$ s3cmd ls 2018-07-09 19:13 s3://container-1 2018-07-09 19:13 s3://public_bucket [student@ceph-admin ~]$ [student@ceph-admin ~]$ s3cmd mb s3://secure_bucket Bucket 's3://secure_bucket/' created
[student@ceph-admin ~]$ s3cmd ls 2018-07-09 19:13 s3://container-1 2018-07-09 19:13 s3://public_bucket 2018-07-09 19:55 s3://secure_bucket [student@ceph-admin ~]$
Congratulations! You've successfully secured your Ceph object storage endpoint using you domain name of your choice and SSL certificates.
Closing
As you can see, acquiring and setting up SSL certificates involves some careful configuration and depends on your chosen CA (how Easy & Fast to acquire SSL certificate). With initiatives like "HTTPS Everywhere," it's no longer just web sites hosting deliverable content that must have SSL; API and service endpoints should also offer encrypted transport.
Note: HTTPS is designed to prevent eavesdropping and Man-in-the-Middle-Attacks. Always practice defense in depth. Multiple layers of security are needed to more fully secure your web site/service endpoints.
About the author
Browse by channel
Automation
The latest on IT automation for tech, teams, and environments
Artificial intelligence
Updates on the platforms that free customers to run AI workloads anywhere
Open hybrid cloud
Explore how we build a more flexible future with hybrid cloud
Security
The latest on how we reduce risks across environments and technologies
Edge computing
Updates on the platforms that simplify operations at the edge
Infrastructure
The latest on the world’s leading enterprise Linux platform
Applications
Inside our solutions to the toughest application challenges
Original shows
Entertaining stories from the makers and leaders in enterprise tech
Products
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Cloud services
- See all products
Tools
- Training and certification
- My account
- Customer support
- Developer resources
- Find a partner
- Red Hat Ecosystem Catalog
- Red Hat value calculator
- Documentation
Try, buy, & sell
Communicate
About Red Hat
We’re the world’s leading provider of enterprise open source solutions—including Linux, cloud, container, and Kubernetes. We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.
Select a language
Red Hat legal and privacy links
- About Red Hat
- Jobs
- Events
- Locations
- Contact Red Hat
- Red Hat Blog
- Diversity, equity, and inclusion
- Cool Stuff Store
- Red Hat Summit