Using Ansible for Fernet Key Rotation on Red Hat OpenStack Platform 11

In our first blog post on the topic of Fernet tokens, we explored what they are and why you should think about enabling them in your OpenStack cloud. In our second post, we looked at the method for enabling these

Fernet tokens in Keystone are fantastic. Enabling these, instead of UUID or PKI tokens, really does make a difference in your cloud's performance and overall ease of management. I get asked a lot about how to manage keys on your controller cluster when using Fernet. As you may imagine, this could potentially take your cloud down if you do it wrong. Let's review what Fernet keys are, as well as how to manage them in your Red Hat OpenStack Platform cloud.

Prerequisites

• A Red Hat OpenStack Platform 11 director-based deployment
• One or more controller nodes
• Git command-line client

What are Fernet Keys?

Fernet keys are used to encrypt and decrypt Fernet tokens in OpenStack's Keystone API. These keys are stored on each controller node, and must be available to authenticate and validate users of the various OpenStack components in your cloud.

Any given implementation of keystone can have (n)keys based on the max_active_keys setting in /etc/keystone/keystone.conf. This number will include all of the types listed below.

There are essentially three types of keys:

Primary

Primary keys are used for token generation and validation. You can think of this as the active key in your cloud. Any time a user authenticates, or is validated by an OpenStack API, these are the keys that will be used. There can only be one primary key, and it must exist on all nodes (usually controllers) that are running the keystone API. The primary key is always the highest indexed key.

Secondary

Secondary keys are only used for token validation. These keys are rotated out of primary status, and thus are used to validate tokens that may exist after a new primary key has been created. There can be multiple secondary keys, the oldest of which will be deleted based on your max_active_keys setting after each key rotation.

Staged

These keys are always the lowest indexed keys (0). Whenever keys are rotated, this key is promoted to a primary key at the highest index allowable by max_active_keys. These keys exist to allow you to copy them to all nodes in your cluster before they're promoted to primary status. This avoids the potential issue where keystone fails to validate a token because the key used to encrypt it does not yet exist in /etc/keystone/fernet-keys.

The following example shows the keys that you'd see in /etc/keystone/fernet-keys, with max_active_keys set to 4.

0 (staged: the next primary key)
1 (primary: token generation & validation)

---

Upon performing a key rotation, our staged key (0), will be the new primary key (2), while our old primary key (1), will be moved to secondary status (1).

0 (staged: the next primary key)
1 (secondary: token validation)
2 (primary: token generation & validation)

---

We have three keys here, so yet another key rotation will produce the following result:

0 (staged: the next primary key)
1 (secondary: token validation)
2 (secondary: token validation)
3 (primary: token generation & validation)

Our staged key (0), now becomes our primary key (3). Our old primary key (2), now becomes a secondary key (2), and (1) remains a secondary key.

We now have four keys, the number we've set in max_active_keys. One more final rotation would produce the following:

0 (staged: the next primary key)
1 (deleted)
2 (secondary: token validation)
3 (secondary: token validation)
4 (primary: token generation & validation)

Our oldest key, secondary (1), is deleted. Our previously staged key (0), is moved to primary (4) status.  A new staged key (0) is created. And finally our old primary key (3) is moved to secondary status.

If you haven't noticed this by now, rotating keys will always remove the key with the lowest index, excluding 0 -- up to your max_active_keys. Additionally, note that you must be careful to set your max_active_keys configuration setting to something that makes sense, given your token lifetime and how often you plan to rotate your keys.

When to rotate?

The answer to this question would probably be different for most organizations. My take on this is simply: if you can do it safely, why not automate it and do it on a regular basis? Your threat model and use-case would normally dictate this or you may need to adhere to certain encryption and key management security controls in a given compliance framework. Whatever the case, I think about regular key rotation as a best-practices security measure. You always want to limit the amount of sensitive data, in this case Fernet tokens, encrypted with a single version of any given encryption key. Rotating your keys on a regular basis creates a smaller exposure surface for your cloud and your users.

How many keys do you need active at one time? This all depends on how often you plan to rotate them, as well as how long your token lifetime is. The answer to this can be expressed in the following equation:

fernet-keys = token-validity(hours) / rotation-time(hours) + 2

Let’s use an example of rotation every 8 hours, with a default token lifetime of 24 hours. This would be

24 hours / 8 hours + 2 = 5

Five keys on your controllers would ensure that you always had an active set of keys for your cloud. With this in mind, let’s look at way to rotate your keys using Ansible.

Rotating Fernet keys

So you may be wondering, how does one automate this process? You can image that this process can be painful and prone to error if done by hand. While you could use the fernet_rotate command to do this on each node manually, why would you?

Let's look at how to do this with Ansible, Red Hat's awesome tool for automation. If you're new to Ansible, please do yourself a favor and check out this quick-start video.

We'll be using an Ansible role, created by my fellow Red Hatter Juan Antonio Osorio (Ozz), one of the coolest guys I know. This is just one way of doing this. For a Red Hat OpenStack Platform install you should contact Red Hat support to review your options and support implications. And of course, your results may vary so be sure to test out on a non-production install!

Let's start by logging into your Red Hat OpenStack director node as the stack user, and creating a roles directory in /home/stack:

$ cat << EOF > ~/rotate.yml
- hosts: controller 
  become: true 
  roles: 
    - tripleo-fernet-keys-rotation
EOF

We need to source our stackrc, as we'll be operating on our controller nodes in the next step

$ source ~/stackrc

Using a dynamic inventory from /usr/bin/tripleo-ansible-inventory, we'll run this playbook and rotate the keys on our controllers

$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory rotate.yml

Ansible Role Analysis

What happened? Looking at Ansible's output, you'll note that several tasks were performed. If you'd like to see these tasks, look no further than /home/stack/roles/tripleo-fernet-keys-rotation/tasks/main.yml:

This task runs a python script, generate_key_yaml.py, in ~/roles/tripleo-ansible-inventory/files, that creates a new fernet key:

- name: Generate new key
 script: generate_key_yaml.py
 register: new_key_register
 run_once: true

This task will take the output of the previous task, from stdout, and register it as the new_key.

- name: Set new key fact
 set_fact:
 new_key: "{{ new_key_register.stdout }}"

Next, we get a sorted list of the keys that currently exist in /etc/keystone/fernet-keys

- name: Get current primary key index
 shell: ls /etc/keystone/fernet-keys | sort -r | head -1
 register: current_key_index_register

Let's set the next primary key index

- name: Set next key index fact
 set_fact:
 next_key_index: "{{ current_key_index_register.stdout|int + 1 }}"

Now we'll move the staged key to the new primary key

- name: Move staged key to new index
 command: mv /etc/keystone/fernet-keys/0 /etc/keystone/fernet-keys/{{ next_key_index }}

Next, let's set our new_key to the new staged key

- name: Set new key as staged key
 copy:
 content: "{{ new_key }}"
 dest: /etc/keystone/fernet-keys/0
 owner: keystone
 group: keystone
 mode: 0600

Finally, we'll reload (not restart) httpd on the controller, allowing keystone to load the new keys

- name: Reload httpd
 service:
 name: httpd
 state: reloaded

Scheduling

Now that we have a way to automate rotation of our keys, it's time to schedule this automation. There are several ways you could do this:

Cron

You could, but why?

Systemd Realtime Timers

Let's create the systemd service that will run our playbook:

cat << EOF > /etc/systemd/system/fernet-rotate.service
[Unit]
Description=Run an Ansible playbook to rotate fernet keys on the overcloud
[Service]
User=stack
Group=stack
ExecStart=/usr/bin/ansible-playbook \
  -i /usr/bin/tripleo-ansible-inventory /home/stack/rotate.yml
EOF

Now we'll create a timer with the same name, only with .timer as the suffix, in /etc/systemd/system on the director node:

cat << EOF > /etc/systemd/system/fernet-rotate.timer
[Unit]
Description=Timer to rotate our Overcloud Fernet Keys weekly
[Timer]
OnCalendar=weekly
Persistent=true
[Install]
WantedBy=timers.target
EOF

Ansible Tower

I like how your thinking! But that’s a topic for another day.

Red Hat OpenStack Platform 12

Red Hat OpenStack Platform 12 provides support for key rotation via Mistral. Learn all about Red Hat OpenStack Platform 12 here.

What about logging?

Ansible to the rescue!

Ansible will use the log_path configuration option from /etc/ansible/ansible.cfg, ansible.cfg in the directory of the playbook, or $HOME/.ansible.cfg. You just need to set this and forget it.

So let's enable this service and timer, and we're off to the races:

$ sudo systemctl enable fernet-rotate.service
$ sudo systemctl enable fernet-rotate.timer

Credit: Many thanks to Lance Bragstad and Dolph Matthews for the key rotation methodology.