An easier way to manage disk decryption at boot with Red Hat Enterprise Linux 7.5 using NBDE

How many times have you had to staff the server room during the graveyard shift just to enter a password to unlock encrypted disks at boot time? Has this requirement kept you away from securing your data? What are your options?

Red Hat has included disk encryption for years with Linux Unified Key Setup-on-disk-format (LUKS). This solution is easy to implement and configure for your encryption needs, but

the management and practicality of its key management is horrible for servers. It requires a passphrase at boot or mount time which has to be entered manually. This makes the solution a headache for system administrators.

Starting in RHEL 7.4, with complete support in RHEL 7.5, Red Hat has implemented an additional component that can be leveraged to enable LUKS disks remotely.  This is called Network Bound Disk Encryption (NBDE). The concept is simple: a RHEL 7.5 client with a LUKS mount makes a remote call to a decryption key server. If the keys match, the mount happens all without human interaction! With growing server deployments in public clouds, easy, secure encryption management is a must.

What are the new components? On the client side (the system with the LUKS mount) we introduce a framework called CLEVIS. On the server side (the system that will do the remote unlocking) we utilize a TANG service.

Let us walk through all the steps to get this working.

1. TANG service setup
2. LUKS volume refresher
3. CLEVIS configuration

TANG (server side) service setup

Because mounting a LUKS volume in an enterprise environment would be critical, we want to configure the TANG service to be highly available. This can be done a number of ways, listed below, in no specific order.

1. DNS round robin - in which you have multiple TANG servers all sharing an A record
2. Utilize an external load balancer - in which you have a load balancer manage the connections to multiple TANG servers
3. Store multiple keys in the LUKS metadata, one for each TANG server.

Below are the steps to make the TANG service highly available by storing multiple keys in the LUKS metadata.

On a pair of RHEL7.5 servers, run the following commands:

[root@tang1 ~]# yum -y install tang
[root@tang2 ~]# yum -y install tang

[root@tang1 ~]# systemctl enable tangd.socket –now
[root@tang2 ~]# systemctl enable tangd.socket –now

[root@tang1 ~]# firewall-cmd --zone=public --add-port=80/tcp –permanent
[root@tang1 ~]# firewall-cmd --reload

[root@tang2 ~]# firewall-cmd --zone=public --add-port=80/tcp –permanent
[root@tang2 ~]# firewall-cmd --reload

The TANG service listens on port TCP 80 by default.

You can override this port in the systemd unit file: tangd.socket

LUKS volume refresher

Below are the steps to create a LUKS volume, start/open a LUKS volume, format the LUKS volume, and mount the LUKS volume.

Create LUKS volume:

[root@beast ~]# cryptsetup --verify-passphrase luksFormat /dev/md0

WARNING!

========

This will overwrite data on /dev/md0 irrevocably.

Are you sure? (Type uppercase yes): YES

Enter passphrase: redhat2018
Verify passphrase: redhat2018

Open encrypted device:

[root@beast ~]# cryptsetup luksOpen /dev/md0 secret
Enter passphrase for /dev/md0: redhat2018

Look at new device:

[root@beast ~]# ls /dev/mapper/secret
/dev/mapper/secret

Format encrypted device:

[root@beast ~]# mkfs.xfs /dev/mapper/secret

Mount encrypted device:

[root@beast ~]# mount /dev/mapper/secret /SECRET

View disk space of encrypted mount:

[root@beast ~]# df /SECRET
Filesystem         1K-blocks  Used Available Use% Mounted on
/dev/mapper/secret 499727972 32944 499695028   1% /SECRET

Tell LUKS to open device on boot – which will prompt for passphrase

[root@beast ~]# echo secret   /dev/md0 none _netdev >> /etc/crypttab

Update fstab before reboot!

/dev/mapper/secret         /SECRET                 xfs         _netdev        1    2

The _netdev option in both the /etc/crypttab and the /etc/fstab configuration files is required for CLEVIS to work with LUKS at boot time.

CLEVIS (client side) configuration

[root@beast ~# yum install clevis clevis-luks clevis-dracut

Before we get too far down the road, let's test the connectivity of our TANG servers.  Download TANG server advertisements:

[root@beast ~]# curl -f http://tang1.i.skinnerlabs.com/adv > adv1.jws
[root@beast ~]# curl -f http://tang2.i.skinnerlabs.com/adv > adv2.jws

Think of the advertisement like a public key. If we can reach this file from each TANG server, we know the CLEVIS service should work properly.

We now need to associate our LUKS volume with each TANG server. You will need to do this step for each TANG server. This step activates a decryption key into the LUKS metadata of the volume, once for each TANG server.

BIND LUKS to TANG Server: TANG1

[root@beast ~]# clevis bind luks -d /dev/md0 tang '{"url":"http://tang1.i.skinnerlabs.com"}'

The advertisement contains the following signing keys:
Mdbv_aFzqDpRR9_L-O-ByY-a9B8

Do you wish to trust these keys? [ynYN] Y

You are about to initialize a LUKS device for metadata storage.
Attempting to initialize it may result in data loss if data was
already written into the LUKS header gap in a different format.
A backup is advised before initialization is performed.

Do you wish to initialize /dev/md0? [yn] y
Enter existing LUKS password:  redhat2018

BIND LUKS to TANG Server: TANG2

[root@beast ~]# clevis bind luks -d /dev/md0 tang '{"url":"http://tang2.i.skinnerlabs.com"}'

The advertisement contains the following signing keys:
kFH77GdVfZ11CRfEQ5U47w3jGfQ

Do you wish to trust these keys? [ynYN] Y
Enter existing LUKS password:  redhat2018

We can confirm that both TANG servers have keys registered to our LUKS volume by using the luksmeta user tool. It will display all keys registered to a volume. Slot 0 is actually our passphrase of redhat2018. You can see that it is active. This means that if both our TANG servers are offline, we can still rely on our manually entered passphrase.

[root@beast ~]# yum -y install luksmeta
[root@beast ~]# luksmeta show -d /dev/md0
0   active empty
1   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
2   active cb6e8904-81ff-40da-a84a-07ab9ab5715e
3 inactive empty
4 inactive empty
5 inactive empty
6 inactive empty
7 inactive empty

The final step is to determine what type of boot unlocking we need. If the LUKS volume is a ROOT volume, an updated initramfs is required. Since CLEVIS relies on networking to connect to the TANG server, you can either configure dracut networking use DHCP or you can statically assign IP address information. Both steps are laid out below:

ROOT Volumes need to update initramfs:

1. DHCP ENVIRONMENT

[root@beast ~]# dracut -f

2. NON DHCP ENVIRONMENTS

[root@beast ~]# dracut -f --kernel-cmdline "ip=192.168.33.225 netmask=255.255.255.0 gateway=192.168.33.1 nameserver=192.168.33.45"

NON-ROOT LUKS volumes need a helper app.  Simply enable the service as outlined below:

[root@beast ~]# systemctl enable clevis-luks-askpass.path

At this point, you need to test. Reboot the server that has the LUKS volume and tail your TANG service logs. Once the CLEVIS dracut/helper application requests a decryption key, you should see the TANG server log the request.

TANG Logs:

[root@tang2 log]# tail -f messages
Mar 19 12:53:32 tang2 systemd: Created slice system-tangd.slice.
Mar 19 12:53:32 tang2 systemd: Starting system-tangd.slice.
Mar 19 12:53:32 tang2 systemd: Started Tang Server (192.168.33.225:42312).
Mar 19 12:53:32 tang2 systemd: Starting Tang Server (192.168.33.225:42312)...
Mar 19 12:53:32 tang2 tangd: 192.168.33.225 POST /rec/9mbH8oDHppKTzwmD_b8EsfbZXFI => 200 (src/tangd.c:168)

If everything is configured correctly, the LUKS passphrase prompt will disappear within a few seconds, and the LUKS volume should mount, without any human interaction.

NBDE supports root, non-root and removable LUKS volumes. Now, there is NO excuse to not use encryption at rest on RHEL servers.

More information:

RHEL 7 documentation link for LUKS:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-encryption

RHEL7 documentation link for NBDE:

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-using_network-bound_disk_encryption