HTTPS-ization of Ceph object storage public endpoint

Hypertext Transfer Protocol Secure (HTTPS) is the secure version of HTTP, uses encrypted communication between the user and the server. HTTPS avoids Man-in-the-Middle-Attack attacks by relying on Secure Socket Layer (SSL) and Transport Layer Security (TLS) protocols to establish an encrypted connection to shuttle data securely between a client and a server.

This blog post takes you step by step through the process of adding SSL/TLS security to Ceph object storage endpoints. Ceph is a scalable, open-source object storage solution that provides Amazon S3 and SWIFT compatible APIs you can use to build your own public or private cloud object storage solution. Learn more about Ceph at Red Hat Ceph Storage.

Setting up domain record sets

Let's begin by setting up a domain name and configuring its record sets. If you already have a domain name you'd like to use for the Ceph endpoint, great. If not, you can purchase one from any domain registrar. In this example, we'll use the domain name ceph-s3.com.

First, we need to add record sets to the domain such that the domain name can resolve into the IP address of the Ceph RADOS Gateway or the load balancer (LB) that is fronting your Ceph RADOS Gateways. To do this, you need to log in to your domain registrar dashboard and create two record sets:

1. A type recordset with Domain Name and IPv4 Address of your Internet-accessible LB or Ceph RGW host. (If you're using IPv6, choose AAAA type.)
2. CNAME type recordset Wildcard Subdomain Name and IPv4 Address of your Internet-accessible LB or Ceph RGW host

Note: The reason we chose the wildcard subdomain name is important. We want to resolve all subdomains (e.g., bucket.ceph-s3.com) to the same IP address, because S3 treats subdomain prefixes as the bucket name.

Domain record set additions are highlighted in the following screenshot:

DNS changes usually take a few tens of minutes to propagate, once the changes are synced. We should be able to ping domain name as well as any subdomain from anywhere on the Internet, provided ICMP Ingress traffic is allowed on the host.

karasing-OSX:~$ ping ceph-s3.com
PING ceph-s3.com (54.209.4.196): 56 data bytes
64 bytes from 54.209.4.196: icmp_seq=0 ttl=40 time=1095.231 ms
64 bytes from 54.209.4.196: icmp_seq=1 ttl=40 time=1099.570 ms
64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1199.266 ms
64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1199.266 ms
^C
--- ceph-s3.com ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1095.231/1131.356/1199.266/48.053 ms
karasing-OSX:~$

karasing-OSX:~$ ping anynameintheuniverse.ceph-s3.com
PING 54.209.4.196 (54.209.4.196): 56 data bytes
64 bytes from 54.209.4.196: icmp_seq=0 ttl=40 time=1491.105 ms
64 bytes from 54.209.4.196: icmp_seq=1 ttl=40 time=1262.021 ms
64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1205.943 ms
64 bytes from 54.209.4.196: icmp_seq=2 ttl=40 time=1205.943 ms
^C
--- 54.209.4.196 ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 1205.943/1319.690/1491.105/123.352 ms
karasing-OSX:~$

SSL certificate: Installation and setup

A SSL certificate is a set of encrypted files that binds an organization's identity, domain name, IP address, and cryptographic keys. Once these SSL certificates are installed on the host server, a secure connection is allowed between the host server and the client machine. In a client’s Internet browser, a green padlock will appear next to the URL as a visual cue to users that traffic is protected.

Some organizations might desire a more advanced certificate that requires additional validation. These SSL certificates must be purchased from a trusted Certificate Authority (CA). For the sake of demonstration in this example, we'll use Let's Encrypt which is a certificate authority that provides free X.509 certificates for Transport Layer Security (TLS) encryption. [ credits: [wikipedia](https://en.wikipedia.org/wiki/Let's_Encrypt) & thanks Let's Encrypt for your free service]

Next, we'll install epel-release and certbot, a CLI tool for requesting SSL certificates, from Let's Encrypt CA.

yum install -y http://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install -y certbot

Request for SSL certificate using certbot CLI client

certbot certonly --manual -d ceph-s3.com -d *.ceph-s3.com --agree-tos
manual-public-ip-logging-ok --preferred-challenges dns-01 --server
https://acme-v02.api.letsencrypt.org/directory

Note: The first -d option in certbot CLI represent base domain, the subsequent -d options represents sub-domains.

Important: Make sure you are using wildcard (*) for subdomain, because we are requesting a wildcard subdomain SSL certification from Let’s Encrypt. If a certificate is only issued for base domain,  it will not be compatible with subdomain prefix notation.

The following snippet shows the output of certbot CLI command. The DNS challenge method will generate two DNS TXT records, which must be added as a TXT record set for your domain (from the domain registrar dashboard):

[root@ceph-admin ~]# certbot certonly --manual -d ceph-s3.com -d *.ceph-s3.com --agree-tos
--manual-public-ip-logging-ok --preferred-challenges dns-01 --server
https://acme-v02.api.letsencrypt.org/directory
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for ceph-s3.com
dns-01 challenge for ceph-s3.com

-------------------------------------------------------------------------------
Please deploy a DNS TXT record under the name
_acme-challenge.ceph-s3.com with the following value:

BzL-LXXkDWwdde8RFUnbQ3fdYt5N6ZXELu4T26KIXa4   <== This Value

Before continuing, verify the record is deployed.

-------------------------------------------------------------------------------
Press Enter to continue

-------------------------------------------------------------------------------
Please deploy a DNS TXT record under the name
_acme-challenge.ceph-s3.com with the following value:

O-_g-eeu4cSI0xXSdrw3OBrWVgzZXJC59Xjkhyk39MQ    <== This Value

Before continuing, verify the record is deployed.

-------------------------------------------------------------------------------

Keep the certbot CLI command running, and open your domain registrar dashboard. Deploy a new DNS TXT record under the name _acme-challenge.yourdomain.com, and enter both these values (marked with <== in the preceding output) in two separate lines inside double quotes. The following screenshot shows how:

Keep the certbot CLI command running and, once you've successfully added DNS TXT record in your domain record set, open another terminal. Now, we'll verify if DNS TXT records are applied on your domain by running the command host -t txt _acme-challenge.yourdomain.com. You should be able to see the same DNS TXT as mentioned in certbot CLI. If you do not, wait for a few minutes for DNS synchronization to occur.

karasing-OSX:~$ host -t txt _acme-challenge.ceph-s3.com
_acme-challenge.ceph-s3.com descriptive text
"BzL-LXXkDWwdde8RFUnbQ3fdYt5N6ZXELu4T26KIXa4"
_acme-challenge.ceph-s3.com descriptive text
"O-_g-eeu4cSI0xXSdrw3OBrWVgzZXJC59Xjkhyk39MQ"
karasing-OSX:~$

Once you've verified DNS TXT records are applied to your domain, return to certbot CLI and press Enter to continue. You will be notified that the system is waiting for verification and cleaning up challenges. You should then see the following message:

Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/ceph-s3.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/ceph-s3.com/privkey.pem
Your cert will expire on 2018-10-07. To obtain a new or tweaked
version of this certificate in the future, simply run certbot
again. To non-interactively renew *all* of your certificates, run
"certbot renew"

If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
Donating to EFF:                    https://eff.org/donate-le

[root@ceph-admin ~]#

At this point, SSL certificates have been issued to your domain/host, so now we must verify them.

[root@ceph-admin ~]# ls -l /etc/letsencrypt/live/ceph-s3.com/
total 4
lrwxrwxrwx 1 root root  35 Jul 9 19:06 cert.pem -> ../../archive/ceph-s3.com/cert1.pem
lrwxrwxrwx 1 root root  36 Jul 9 19:06 chain.pem -> ../../archive/ceph-s3.com/chain1.pem
lrwxrwxrwx 1 root root  40 Jul 9 19:06 fullchain.pem -> ../../archive/ceph-s3.com/fullchain1.pem
lrwxrwxrwx 1 root root  38 Jul 9 19:06 privkey.pem -> ../../archive/ceph-s3.com/privkey1.pem
-rw-r--r-- 1 root root 682 Jul  9 19:06 README
[root@ceph-admin ~]#

Installing LB for object storage service

Many like HAProxy, because it's easy and does its job well. In this example, we will use HAProxy to perform SSL termination for our domain name (Ceph object storage endpoint).

Note: Starting with Red Hat Ceph Storage 2, Ceph RADOS Gateway natively supports TLS by relying on OpenSSL Library. You can get more information on native SSL/TLS configuration here. In this example, we specifically choose to terminate SSL at HAProxy level. This gives us an advantage like when we have multiple instances of Ceph RGW we do not need to get multiple domain names/SSL certificates for each of them. One domain name with SSL termination at LB does the job.

Next, we'll install HAproxy on the same host whose public IP has been bound with your domain name, in this case, ceph-s3.com.

yum install -y haproxy

Next, we'll create a certs directory.

mkdir -p /etc/haproxy/certs

Combine certificate files fullchain.pem and privkey.pem into a single file for our domain next.

DOMAIN='ceph-s3.com' sudo -E bash -c 'cat /etc/letsencrypt/live/$DOMAIN/fullchain.pem
/etc/letsencrypt/live/$DOMAIN/privkey.pem > /etc/haproxy/certs/$DOMAIN.pem'

The next step is to change the permission of the certs directory.

chmod -R go-rwx /etc/haproxy/certs

Optionally, you can move the haproxy.cfg original file and create a new config file with the following configuration settings:

mv /etc/haproxy/haproxy.cfg /etc/haproxy/haproxy.cfg.orig
vim /etc/haproxy/haproxy.cfg

global
    log        127.0.0.1 local2
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy
    daemon
    tune.ssl.default-dh-param 2048     
    stats socket /var/lib/haproxy/stats
defaults
    mode                    http
    log                    global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    option httpchk HEAD /
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
frontend www-http
   bind 10.100.0.10:80
   reqidel                      ^XForwardedFor:.*
   reqadd X-Forwarded-Proto:\ http
   default_backend www-backend
   option  forwardfor
frontend www-https
   bind 10.100.0.10:443 ssl crt /etc/haproxy/certs/ceph-s3.com.pem
   reqadd X-Forwarded-Proto:\ https
   acl letsencrypt-acl path_beg /.well-known/acme-challenge/
   use_backend letsencrypt-backend if letsencrypt-acl
   default_backend www-backend
backend www-backend
   redirect scheme https if !{ ssl_fc }
   server ceph-admin 10.100.0.10:8081 check inter 2000 rise 2 fall 5
backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321

As you may have noted, we're using a couple of non-default parameters in the haproxy config file, such as:

• tune.ssl.default-dh-param is required to provide OpenSSL the necessary parameters for the SSL/TLS handshake.
• frontend www-http binds haproxy to port 80 of the local machine and redirects traffic to the default backend www-backend. If a user uses HTTP protocol in the request, it should redirect to HTTPS.
• frontend www-https binds haproxy to port 443 of the local machine. It also redirects traffic to the default backend www-backend and uses the SSL certificate path to encrypt/terminate the traffic.
• frontend www-https also uses letsencrypt-backend if you want to auto-renew the SSL certificate from Let's Encrypt CA.
• backend www-backend simply redirects all the SSL terminated traffic to Ceph RGW node 10.100.0.10:8081. For HA and performance, you must have multiple Ceph RGW instances whose IPs should be added in the same backend section so that HAProxy can load balance among Ceph RGW instances.

Finally, we restart HAproxy and verify its listening on ports 80 and 443:

systemctl start haproxy
systemctl status haproxy ; netstat -plunt | grep -i haproxy

Configuring Ceph RGW

Up to this point, we've configured the domain name, set required record sets, generated SSL certificate, configured HAProxy to encrypt/terminate SSL, and redirected traffic to the Ceph RGW instance. Now we need to configure Ceph RGW to listen on port 8081 as configured in HAproxy. To do so, in the Ceph RGW node edit /etc/ceph/ceph.conf and update the client.rgw section as shown in the following:

[client.rgw.ceph-admin]
host = ceph-admin
keyring = /var/lib/ceph/radosgw/ceph-rgw.ceph-admin/keyring
log file = /var/log/ceph/ceph-rgw-ceph-admin.log
rgw frontends = civetweb port=10.100.0.10:8081 num_threads=512
rgw resolve cname = true
rgw dns name = ceph-s3.com

Note: rgw resolve cname = true forces rgw to use the DNS CNAME record of the request hostname field (if the hostname is not equal to rgw dns name).

Note: rgw dns name = ceph-s3.com is the DNS name of the served domain.

Now, we'll restart the Ceph RGW instance and verify its listening on port 8081.

systemctl restart ceph-radosgw@rgw.ceph-admin.service
netstat -plunt | grep -i rados

Accessing Ceph object storage secure endpoint

To test the HTTPS-enabled Ceph object storage URL, execute the following curl command or type https://ceph-s3.com in any web browser:

curl https://ceph-s3.com

It should yield output like the following:

[student@ceph-admin ~]$ curl https://ceph-s3.com
anonymous
[student@ceph-admin ~]$
[student@ceph-admin ~]$

Let's try accessing Ceph object storage using S3cmd:

yum install -y s3cmd

Configure S3cmd CLI by providing config options like access/secret keys, Ceph S3 secure endpoints in host/host-bucket parameters.

s3cmd --access_key=S3user1 --secret_key=S3user1key --host=ceph-s3.com
--host-bucket="%(bucket)s.ceph-s3.com" --dump-config > /home/student/.s3cfg

Note: By default, s3cmd uses HTTPS connection, so there is no need to explicitly specify that.

Next, we'll interact with Ceph object storage using s3cmd ls, s3cmd mb commands:

[student@ceph-admin ~]$ s3cmd ls
2018-07-09 19:13  s3://container-1
2018-07-09 19:13  s3://public_bucket
[student@ceph-admin ~]$
[student@ceph-admin ~]$ s3cmd mb s3://secure_bucket
Bucket 's3://secure_bucket/' created

[student@ceph-admin ~]$ s3cmd ls
2018-07-09 19:13  s3://container-1
2018-07-09 19:13  s3://public_bucket
2018-07-09 19:55  s3://secure_bucket
[student@ceph-admin ~]$

Congratulations! You've successfully secured your Ceph object storage endpoint using you domain name of your choice and SSL certificates.

Closing

As you can see, acquiring and setting up SSL certificates involves some careful configuration and depends on your chosen CA (how Easy & Fast to acquire SSL certificate). With initiatives like "HTTPS Everywhere," it's no longer just web sites hosting deliverable content that must have SSL; API and service endpoints should also offer encrypted transport.

Note: HTTPS is designed to prevent eavesdropping and Man-in-the-Middle-Attacks. Always practice defense in depth. Multiple layers of security are needed to more fully secure your web site/service endpoints.