Hardening Gluster Installations with TLS

Data availability, confidentiality, and integrity are important aspects of security. One concern is securing the data and control information used by Gluster nodes which flows over the network. To address this problem network encryption can now be enabled in Gluster using TLS. Encrypting communications between glusterd, Gluster client, and the Gluster server will add significant complexity to any attacker attempting to abuse the Gluster nodes and services.

Deployment guidelines recommend that Gluster’s internal network should be isolated and not accessible to general users. Even with an isolated network, requiring these connections to use cryptography can prevent access to in-flight network data from attackers with access to these internal networks.

Enabling encryption requires an attacker to be able to successfully authenticate before being able to interact with Gluster nodes and services.

Listed below are the two types of encryption levels available in Gluster:

Caveats

TLS only provides authentication and encryption; it does not provide authorization. Once a Glusterfs node or client is connected then there are no further checks for authorization or restricting any user from a specific operation. GlusterFS uses TLS-authenticated identities to authorize clients to access Gluster volumes. 

Enabling TLS encryption on Gluster volumes which are in production without encryption will require the server to be restarted, and Gluster clients have to remount to support encryption between Gluster server and client. To use TLS in GlusterFS both sides should negotiate as there are no mixed mode connections; if one side is configured to use TLS the other side can only communicate using TLS encryption otherwise connections will be rejected. No fallback mode prioritizes security rather than availability of the service, as using fallback mode will be insecure and make it prone to downgrading security by not using encryption.

Enabling TLS on GlusterFS

Enabling TLS on GlusterFS using, for example, 3 Gluster server nodes and 1 client with fresh installation is a relatively easy task:

Using Gluster nodes to provide Gluster volume secvol01 with 3 replicas:

• rhgs-sr01 (brick_01)
• rhgs-sr02 (brick_01)
• rhgs-sr03 (brick_01)
• client-01

Following files are required to enable TLS:

1.  glusterfs.key :  A unique private key to generate certificate signing requests and decrypting traffic. This key must not be shared and should be kept protected on the Gluster node.

2. glusterfs.pem:  TLS certificate for the Gluster node, this certificate should not be shared outside the trust boundary.

3. glusterfs.ca : Certificate Authority's (CA) certificates. These are the certificates which were used to sign certificate signing requests for Gluster node.

4. secure-access : enables management encryption which are glusterd connections between all servers in trusted storage pool, only required for management encryption. 

The glusterfs.key file needs to be secured, as certificates are public so anyone can claim identity if they get access to the private key. 

The certificate contains the identity of the owner’s Principal + Certificate Authority CA. If the certificate is signed by common CA then the principal and CA identities are different. If certificates are self-signed then these identities are same.

For authentication, the client's certificates also need to be part of the server's certificate and vice-versa. For the same reason glusterfs.pem files from all the servers are concatenated and made available to clients. As a client won’t have access to private keys of Gluster servers, clients cannot claim identity of the server.

The files glusterfs.key, glusterfs.pem. and glusterfs.ca should be placed in the /etc/ssl directory. By default glusterfs looks for these files in /etc/ssl directory. These locations can only be changed in the volfile, not from glusterfs-cli. 

...
#if !defined(DEFAULT_ETC_SSL)
#ifdef GF_LINUX_HOST_OS
#define DEFAULT_ETC_SSL "/etc/ssl"
#endif
#ifdef GF_BSD_HOST_OS
#define DEFAULT_ETC_SSL "/etc/openssl"
#endif
#ifdef GF_DARWIN_HOST_OS
#define DEFAULT_ETC_SSL "/usr/local/etc/openssl"
#endif
#if !defined(DEFAULT_ETC_SSL)
#define DEFAULT_ETC_SSL "/etc/ssl"
#endif
#endif

#if !defined(DEFAULT_CERT_PATH)
#define DEFAULT_CERT_PATH DEFAULT_ETC_SSL "/glusterfs.pem"
#endif
#if !defined(DEFAULT_KEY_PATH)
#define DEFAULT_KEY_PATH DEFAULT_ETC_SSL "/glusterfs.key"
#endif
#if !defined(DEFAULT_CA_PATH)
#define DEFAULT_CA_PATH DEFAULT_ETC_SSL "/glusterfs.ca"
#endif
...

Generating keys, signing certificates, and setting up the service

The following steps must be done on each Gluster node and client to enable encryption within trusted pool and in between client and Gluster nodes. All files should be kept in the /etc/ssl directory.

Generate private keys for each node and client:

~]# openssl genrsa -out /etc/ssl/glusterfs.key 4096

Create self-signed certificate from generated private key:

~]# openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=`hostname -f`" -out /etc/ssl/glusterfs.pem

In the steps above, hostname -f would be the FQDN. It can also be hostname, or the IP Address of machine.

Concatenate all glusterfs.pem files from servers and client node to the gluster.ca file and copy the glusterfs.ca file to/etc/ssl/glusterfs.ca on rhgs-sr01, rhgs-sr02, rhgs-sr03, client-01.  Concatenation of certificates is required if using self-signed certs.

Using self-signed certificates is not a scalable solution and is a tedious task. If there is common CA then Certificate Signing Request (CSR) can be generated and sent to intermediate CA or signing authority to get signed certificate. To generate a CSR the following command needs to be issued: 

~]$ openssl req -new -sha256 -key /etc/ssl/glusterfs.key \

    -subj "/CN=`hostname -f`" -out `hostname -f`.csr

Management Encryption

Management encryption is enabled by creating a secure-access file on each machine. This file should be created on all machines. This file enables TLS encryption on glusterd connections. The same is required on client machine to communicate with glusterd when mounting. In older releases secure-access file did not contain anything, but in new gluster releases there have been few code changes to configure TLS. After applying TLS secure-access patch and other patches related to TLS it requires setting transport.socket.ssl-cert-depth in secure-access file.

libglusterfs/src/graph.c

int glusterfs_read_secure_access_file (void)
{
…
 int   cert_depth = 1;   /* Default SSL CERT DEPTH */
…
 char *key = {"^option transport.socket.ssl-cert-depth"};
…
fp = fopen (SECURE_ACCESS_FILE, "r");
…
      if (cert_depth == 0)
                                     cert_depth = 1; /* Default SSL CERT DEPTH */
                             break;
                     }
...

~]# touch /var/lib/glusterd/secure-access
~]# vi /var/lib/glusterd/secure-access
…
option transport.socket.ssl-cert-depth n 
…
# where n is numbers 1,2,3 to set to which depth certificate is to be verified. By default gluster code uses ‘option transport.socket.ssl-cert-depth 1’ which is for self signed certificates.
 
~]# systemctl start glusterd.service

I/O Encryption

To enable I/O encryption between the gluster server and clients follow the procedure as listed below.

Create volume

~]# gluster volume create securevol01 replica 3 transport tcp 
    rhgs-sr01:/brick_01/data \
    rhgs-sr02:/brick_01/data \
    rhgs-sr03:/brick_01/data

Enable TLS on I/O Path

~]# gluster volume set securevol01 client.ssl on

~]# gluster volume set securevol01 server.ssl on

Authorize which identities should be allowed to access volume

The auth.ssl-allow option is used to enable TLS authentication. It accepts a list of common names of server and clients which are allowed to access Gluster volume.

~]# gluster volume set securevol01 auth.ssl-allow rhgs-sr01,rhgs-sr02,rhgs-sr03,client-01

~]# gluster volume set securevol01 ssl.cipher-list 'HIGH:!SSLv2'

After configuring TLS, start the Gluster volume.

~]# gluster volume start securevol01

Verify TLS Connection

>~]# grep SSL /var/log/glusterfs/glusterd.log

~]# openssl s_client -showcerts -connect rhgs-sr02:24007​

On Client

This process is similar to TLS configuration on glusterfs server nodes, copy glusterfs.key, glusterfs.pem, and glusterfs.ca file to /etc/ssl of client. The glusterfs.ca file for client must contain certificate of signing CA, if the certificates are self-signed then glusterfs.ca is a concatenation of certificate files of each glusterfs server.

~]# touch /var/lib/glusterd/secure-access
~]# systemctl start glusterd.service
~]# mount -t glusterfs rhgs-sr01:/securevol01 /mnt/glusterfs

Epilogue

Encrypting traffic between glusterd, Gluster servers, and clients helps to mitigate the problem of MITM and hardens a Gluster installation enough to stop an attacker from accessing Gluster volumes. All files in /etc/ssl should be secured, and in the event of a compromise of server or client their keys and certificates must be revoked, regenerated, and re-signed to re-enable security features.

References

• Red Hat Storage Administration Guide – Network Encryption
• Red Hat Storage Administration Guide - Network Encryption, New Pool
• Gluster Administrator Guide