In our first blog post on the topic of Fernet tokens, we explored what they are and why you should think about enabling them in your OpenStack cloud. In our second post, we looked at the method for enabling these
Fernet tokens in Keystone are fantastic. Enabling these, instead of UUID or PKI tokens, really does make a difference in your cloud's performance and overall ease of management. I get asked a lot about how to manage keys on your controller cluster when using Fernet. As you may imagine, this could potentially take your cloud down if you do it wrong. Let's review what Fernet keys are, as well as how to manage them in your Red Hat OpenStack Platform cloud.
Prerequisites
- A Red Hat OpenStack Platform 11 director-based deployment
- One or more controller nodes
- Git command-line client
What are Fernet Keys?
Fernet keys are used to encrypt and decrypt Fernet tokens in OpenStack's Keystone API. These keys are stored on each controller node, and must be available to authenticate and validate users of the various OpenStack components in your cloud.
Any given implementation of keystone can have (n)keys based on the max_active_keys setting in /etc/keystone/keystone.conf. This number will include all of the types listed below.
There are essentially three types of keys:
Primary
Primary keys are used for token generation and validation. You can think of this as the active key in your cloud. Any time a user authenticates, or is validated by an OpenStack API, these are the keys that will be used. There can only be one primary key, and it must exist on all nodes (usually controllers) that are running the keystone API. The primary key is always the highest indexed key.
Secondary
Secondary keys are only used for token validation. These keys are rotated out of primary status, and thus are used to validate tokens that may exist after a new primary key has been created. There can be multiple secondary keys, the oldest of which will be deleted based on your max_active_keys setting after each key rotation.
Staged
These keys are always the lowest indexed keys (0). Whenever keys are rotated, this key is promoted to a primary key at the highest index allowable by max_active_keys. These keys exist to allow you to copy them to all nodes in your cluster before they're promoted to primary status. This avoids the potential issue where keystone fails to validate a token because the key used to encrypt it does not yet exist in /etc/keystone/fernet-keys.
The following example shows the keys that you'd see in /etc/keystone/fernet-keys, with max_active_keys set to 4.
0 (staged: the next primary key)
1 (primary: token generation & validation)
---
Upon performing a key rotation, our staged key (0), will be the new primary key (2), while our old primary key (1), will be moved to secondary status (1).
0 (staged: the next primary key)
1 (secondary: token validation)
2 (primary: token generation & validation)
---
We have three keys here, so yet another key rotation will produce the following result:
0 (staged: the next primary key)
1 (secondary: token validation)
2 (secondary: token validation)
3 (primary: token generation & validation)
Our staged key (0), now becomes our primary key (3). Our old primary key (2), now becomes a secondary key (2), and (1) remains a secondary key.
We now have four keys, the number we've set in max_active_keys. One more final rotation would produce the following:
0 (staged: the next primary key)
1 (deleted)
2 (secondary: token validation)
3 (secondary: token validation)
4 (primary: token generation & validation)
Our oldest key, secondary (1), is deleted. Our previously staged key (0), is moved to primary (4) status. A new staged key (0) is created. And finally our old primary key (3) is moved to secondary status.
If you haven't noticed this by now, rotating keys will always remove the key with the lowest index, excluding 0 -- up to your max_active_keys. Additionally, note that you must be careful to set your max_active_keys configuration setting to something that makes sense, given your token lifetime and how often you plan to rotate your keys.
When to rotate?
The answer to this question would probably be different for most organizations. My take on this is simply: if you can do it safely, why not automate it and do it on a regular basis? Your threat model and use-case would normally dictate this or you may need to adhere to certain encryption and key management security controls in a given compliance framework. Whatever the case, I think about regular key rotation as a best-practices security measure. You always want to limit the amount of sensitive data, in this case Fernet tokens, encrypted with a single version of any given encryption key. Rotating your keys on a regular basis creates a smaller exposure surface for your cloud and your users.
How many keys do you need active at one time? This all depends on how often you plan to rotate them, as well as how long your token lifetime is. The answer to this can be expressed in the following equation:
fernet-keys = token-validity(hours) / rotation-time(hours) + 2
Let’s use an example of rotation every 8 hours, with a default token lifetime of 24 hours. This would be
24 hours / 8 hours + 2 = 5
Five keys on your controllers would ensure that you always had an active set of keys for your cloud. With this in mind, let’s look at way to rotate your keys using Ansible.
Rotating Fernet keys
So you may be wondering, how does one automate this process? You can image that this process can be painful and prone to error if done by hand. While you could use the fernet_rotate command to do this on each node manually, why would you?
Let's look at how to do this with Ansible, Red Hat's awesome tool for automation. If you're new to Ansible, please do yourself a favor and check out this quick-start video.
We'll be using an Ansible role, created by my fellow Red Hatter Juan Antonio Osorio (Ozz), one of the coolest guys I know. This is just one way of doing this. For a Red Hat OpenStack Platform install you should contact Red Hat support to review your options and support implications. And of course, your results may vary so be sure to test out on a non-production install!
Let's start by logging into your Red Hat OpenStack director node as the stack user, and creating a roles directory in /home/stack:
$ cat << EOF > ~/rotate.yml - hosts: controller become: true roles: - tripleo-fernet-keys-rotation EOF
We need to source our stackrc, as we'll be operating on our controller nodes in the next step
$ source ~/stackrc
Using a dynamic inventory from /usr/bin/tripleo-ansible-inventory, we'll run this playbook and rotate the keys on our controllers
$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory rotate.yml
Ansible Role Analysis
What happened? Looking at Ansible's output, you'll note that several tasks were performed. If you'd like to see these tasks, look no further than /home/stack/roles/tripleo-fernet-keys-rotation/tasks/main.yml:
This task runs a python script, generate_key_yaml.py, in ~/roles/tripleo-ansible-inventory/files, that creates a new fernet key:
- name: Generate new key script: generate_key_yaml.py register: new_key_register run_once: true This task will take the output of the previous task, from stdout, and register it as the new_key. - name: Set new key fact set_fact: new_key: "{{ new_key_register.stdout }}"
Next, we get a sorted list of the keys that currently exist in /etc/keystone/fernet-keys
- name: Get current primary key index shell: ls /etc/keystone/fernet-keys | sort -r | head -1 register: current_key_index_register
Let's set the next primary key index
- name: Set next key index fact set_fact: next_key_index: "{{ current_key_index_register.stdout|int + 1 }}"
Now we'll move the staged key to the new primary key
- name: Move staged key to new index command: mv /etc/keystone/fernet-keys/0 /etc/keystone/fernet-keys/{{ next_key_index }}
Next, let's set our new_key to the new staged key
- name: Set new key as staged key copy: content: "{{ new_key }}" dest: /etc/keystone/fernet-keys/0 owner: keystone group: keystone mode: 0600
Finally, we'll reload (not restart) httpd on the controller, allowing keystone to load the new keys
- name: Reload httpd service: name: httpd state: reloaded
Scheduling
Now that we have a way to automate rotation of our keys, it's time to schedule this automation. There are several ways you could do this:
Cron
You could, but why?
Systemd Realtime Timers
Let's create the systemd service that will run our playbook:
cat << EOF > /etc/systemd/system/fernet-rotate.service [Unit] Description=Run an Ansible playbook to rotate fernet keys on the overcloud [Service] User=stack Group=stack ExecStart=/usr/bin/ansible-playbook \ -i /usr/bin/tripleo-ansible-inventory /home/stack/rotate.yml EOF
Now we'll create a timer with the same name, only with .timer as the suffix, in /etc/systemd/system on the director node:
cat << EOF > /etc/systemd/system/fernet-rotate.timer [Unit] Description=Timer to rotate our Overcloud Fernet Keys weekly [Timer] OnCalendar=weekly Persistent=true [Install] WantedBy=timers.target EOF
Ansible Tower
I like how your thinking! But that’s a topic for another day.
Red Hat OpenStack Platform 12
Red Hat OpenStack Platform 12 provides support for key rotation via Mistral. Learn all about Red Hat OpenStack Platform 12 here.
What about logging?
Ansible to the rescue!
Ansible will use the log_path configuration option from /etc/ansible/ansible.cfg, ansible.cfg in the directory of the playbook, or $HOME/.ansible.cfg. You just need to set this and forget it.
So let's enable this service and timer, and we're off to the races:
$ sudo systemctl enable fernet-rotate.service $ sudo systemctl enable fernet-rotate.timer
Credit: Many thanks to Lance Bragstad and Dolph Matthews for the key rotation methodology.
Sobre el autor
Navegar por canal
Automatización
Las últimas novedades en la automatización de la TI para los equipos, la tecnología y los entornos
Inteligencia artificial
Descubra las actualizaciones en las plataformas que permiten a los clientes ejecutar cargas de trabajo de inteligecia artificial en cualquier lugar
Nube híbrida abierta
Vea como construimos un futuro flexible con la nube híbrida
Seguridad
Vea las últimas novedades sobre cómo reducimos los riesgos en entornos y tecnologías
Edge computing
Conozca las actualizaciones en las plataformas que simplifican las operaciones en el edge
Infraestructura
Vea las últimas novedades sobre la plataforma Linux empresarial líder en el mundo
Aplicaciones
Conozca nuestras soluciones para abordar los desafíos más complejos de las aplicaciones
Programas originales
Vea historias divertidas de creadores y líderes en tecnología empresarial
Productos
- Red Hat Enterprise Linux
- Red Hat OpenShift
- Red Hat Ansible Automation Platform
- Servicios de nube
- Ver todos los productos
Herramientas
- Training y Certificación
- Mi cuenta
- Soporte al cliente
- Recursos para desarrolladores
- Busque un partner
- Red Hat Ecosystem Catalog
- Calculador de valor Red Hat
- Documentación
Realice pruebas, compras y ventas
Comunicarse
- Comuníquese con la oficina de ventas
- Comuníquese con el servicio al cliente
- Comuníquese con Red Hat Training
- Redes sociales
Acerca de Red Hat
Somos el proveedor líder a nivel mundial de soluciones empresariales de código abierto, incluyendo Linux, cloud, contenedores y Kubernetes. Ofrecemos soluciones reforzadas, las cuales permiten que las empresas trabajen en distintas plataformas y entornos con facilidad, desde el centro de datos principal hasta el extremo de la red.
Seleccionar idioma
Red Hat legal and privacy links
- Acerca de Red Hat
- Oportunidades de empleo
- Eventos
- Sedes
- Póngase en contacto con Red Hat
- Blog de Red Hat
- Diversidad, igualdad e inclusión
- Cool Stuff Store
- Red Hat Summit