In our first blog post on the topic of Fernet tokens, we explored what they are and why you should think about enabling them in your OpenStack cloud. In our second post, we looked at the method for enabling these
Fernet tokens in Keystone are fantastic. Enabling these, instead of UUID or PKI tokens, really does make a difference in your cloud's performance and overall ease of management. I get asked a lot about how to manage keys on your controller cluster when using Fernet. As you may imagine, this could potentially take your cloud down if you do it wrong. Let's review what Fernet keys are, as well as how to manage them in your Red Hat OpenStack Platform cloud.
Prerequisites
- A Red Hat OpenStack Platform 11 director-based deployment
- One or more controller nodes
- Git command-line client
What are Fernet Keys?
Fernet keys are used to encrypt and decrypt Fernet tokens in OpenStack's Keystone API. These keys are stored on each controller node, and must be available to authenticate and validate users of the various OpenStack components in your cloud.
Any given implementation of keystone can have (n)keys based on the max_active_keys setting in /etc/keystone/keystone.conf. This number will include all of the types listed below.
There are essentially three types of keys:
Primary
Primary keys are used for token generation and validation. You can think of this as the active key in your cloud. Any time a user authenticates, or is validated by an OpenStack API, these are the keys that will be used. There can only be one primary key, and it must exist on all nodes (usually controllers) that are running the keystone API. The primary key is always the highest indexed key.
Secondary
Secondary keys are only used for token validation. These keys are rotated out of primary status, and thus are used to validate tokens that may exist after a new primary key has been created. There can be multiple secondary keys, the oldest of which will be deleted based on your max_active_keys setting after each key rotation.
Staged
These keys are always the lowest indexed keys (0). Whenever keys are rotated, this key is promoted to a primary key at the highest index allowable by max_active_keys. These keys exist to allow you to copy them to all nodes in your cluster before they're promoted to primary status. This avoids the potential issue where keystone fails to validate a token because the key used to encrypt it does not yet exist in /etc/keystone/fernet-keys.
The following example shows the keys that you'd see in /etc/keystone/fernet-keys, with max_active_keys set to 4.
0 (staged: the next primary key)
1 (primary: token generation & validation)
---
Upon performing a key rotation, our staged key (0), will be the new primary key (2), while our old primary key (1), will be moved to secondary status (1).
0 (staged: the next primary key)
1 (secondary: token validation)
2 (primary: token generation & validation)
---
We have three keys here, so yet another key rotation will produce the following result:
0 (staged: the next primary key)
1 (secondary: token validation)
2 (secondary: token validation)
3 (primary: token generation & validation)
Our staged key (0), now becomes our primary key (3). Our old primary key (2), now becomes a secondary key (2), and (1) remains a secondary key.
We now have four keys, the number we've set in max_active_keys. One more final rotation would produce the following:
0 (staged: the next primary key)
1 (deleted)
2 (secondary: token validation)
3 (secondary: token validation)
4 (primary: token generation & validation)
Our oldest key, secondary (1), is deleted. Our previously staged key (0), is moved to primary (4) status. A new staged key (0) is created. And finally our old primary key (3) is moved to secondary status.
If you haven't noticed this by now, rotating keys will always remove the key with the lowest index, excluding 0 -- up to your max_active_keys. Additionally, note that you must be careful to set your max_active_keys configuration setting to something that makes sense, given your token lifetime and how often you plan to rotate your keys.
When to rotate?
The answer to this question would probably be different for most organizations. My take on this is simply: if you can do it safely, why not automate it and do it on a regular basis? Your threat model and use-case would normally dictate this or you may need to adhere to certain encryption and key management security controls in a given compliance framework. Whatever the case, I think about regular key rotation as a best-practices security measure. You always want to limit the amount of sensitive data, in this case Fernet tokens, encrypted with a single version of any given encryption key. Rotating your keys on a regular basis creates a smaller exposure surface for your cloud and your users.
How many keys do you need active at one time? This all depends on how often you plan to rotate them, as well as how long your token lifetime is. The answer to this can be expressed in the following equation:
fernet-keys = token-validity(hours) / rotation-time(hours) + 2
Let’s use an example of rotation every 8 hours, with a default token lifetime of 24 hours. This would be
24 hours / 8 hours + 2 = 5
Five keys on your controllers would ensure that you always had an active set of keys for your cloud. With this in mind, let’s look at way to rotate your keys using Ansible.
Rotating Fernet keys
So you may be wondering, how does one automate this process? You can image that this process can be painful and prone to error if done by hand. While you could use the fernet_rotate command to do this on each node manually, why would you?
Let's look at how to do this with Ansible, Red Hat's awesome tool for automation. If you're new to Ansible, please do yourself a favor and check out this quick-start video.
We'll be using an Ansible role, created by my fellow Red Hatter Juan Antonio Osorio (Ozz), one of the coolest guys I know. This is just one way of doing this. For a Red Hat OpenStack Platform install you should contact Red Hat support to review your options and support implications. And of course, your results may vary so be sure to test out on a non-production install!
Let's start by logging into your Red Hat OpenStack director node as the stack user, and creating a roles directory in /home/stack:
$ cat << EOF > ~/rotate.yml - hosts: controller become: true roles: - tripleo-fernet-keys-rotation EOF
We need to source our stackrc, as we'll be operating on our controller nodes in the next step
$ source ~/stackrc
Using a dynamic inventory from /usr/bin/tripleo-ansible-inventory, we'll run this playbook and rotate the keys on our controllers
$ ansible-playbook -i /usr/bin/tripleo-ansible-inventory rotate.yml
Ansible Role Analysis
What happened? Looking at Ansible's output, you'll note that several tasks were performed. If you'd like to see these tasks, look no further than /home/stack/roles/tripleo-fernet-keys-rotation/tasks/main.yml:
This task runs a python script, generate_key_yaml.py, in ~/roles/tripleo-ansible-inventory/files, that creates a new fernet key:
- name: Generate new key script: generate_key_yaml.py register: new_key_register run_once: true This task will take the output of the previous task, from stdout, and register it as the new_key. - name: Set new key fact set_fact: new_key: "{{ new_key_register.stdout }}"
Next, we get a sorted list of the keys that currently exist in /etc/keystone/fernet-keys
- name: Get current primary key index shell: ls /etc/keystone/fernet-keys | sort -r | head -1 register: current_key_index_register
Let's set the next primary key index
- name: Set next key index fact set_fact: next_key_index: "{{ current_key_index_register.stdout|int + 1 }}"
Now we'll move the staged key to the new primary key
- name: Move staged key to new index command: mv /etc/keystone/fernet-keys/0 /etc/keystone/fernet-keys/{{ next_key_index }}
Next, let's set our new_key to the new staged key
- name: Set new key as staged key copy: content: "{{ new_key }}" dest: /etc/keystone/fernet-keys/0 owner: keystone group: keystone mode: 0600
Finally, we'll reload (not restart) httpd on the controller, allowing keystone to load the new keys
- name: Reload httpd service: name: httpd state: reloaded
Scheduling
Now that we have a way to automate rotation of our keys, it's time to schedule this automation. There are several ways you could do this:
Cron
You could, but why?
Systemd Realtime Timers
Let's create the systemd service that will run our playbook:
cat << EOF > /etc/systemd/system/fernet-rotate.service [Unit] Description=Run an Ansible playbook to rotate fernet keys on the overcloud [Service] User=stack Group=stack ExecStart=/usr/bin/ansible-playbook \ -i /usr/bin/tripleo-ansible-inventory /home/stack/rotate.yml EOF
Now we'll create a timer with the same name, only with .timer as the suffix, in /etc/systemd/system on the director node:
cat << EOF > /etc/systemd/system/fernet-rotate.timer [Unit] Description=Timer to rotate our Overcloud Fernet Keys weekly [Timer] OnCalendar=weekly Persistent=true [Install] WantedBy=timers.target EOF
Ansible Tower
I like how your thinking! But that’s a topic for another day.
Red Hat OpenStack Platform 12
Red Hat OpenStack Platform 12 provides support for key rotation via Mistral. Learn all about Red Hat OpenStack Platform 12 here.
What about logging?
Ansible to the rescue!
Ansible will use the log_path configuration option from /etc/ansible/ansible.cfg, ansible.cfg in the directory of the playbook, or $HOME/.ansible.cfg. You just need to set this and forget it.
So let's enable this service and timer, and we're off to the races:
$ sudo systemctl enable fernet-rotate.service $ sudo systemctl enable fernet-rotate.timer
Credit: Many thanks to Lance Bragstad and Dolph Matthews for the key rotation methodology.
저자 소개
채널별 검색
오토메이션
기술, 팀, 인프라를 위한 IT 자동화 최신 동향
인공지능
고객이 어디서나 AI 워크로드를 실행할 수 있도록 지원하는 플랫폼 업데이트
오픈 하이브리드 클라우드
하이브리드 클라우드로 더욱 유연한 미래를 구축하는 방법을 알아보세요
보안
환경과 기술 전반에 걸쳐 리스크를 감소하는 방법에 대한 최신 정보
엣지 컴퓨팅
엣지에서의 운영을 단순화하는 플랫폼 업데이트
인프라
세계적으로 인정받은 기업용 Linux 플랫폼에 대한 최신 정보
애플리케이션
복잡한 애플리케이션에 대한 솔루션 더 보기
오리지널 쇼
엔터프라이즈 기술 분야의 제작자와 리더가 전하는 흥미로운 스토리
제품
- Red Hat Enterprise Linux
- Red Hat OpenShift Enterprise
- Red Hat Ansible Automation Platform
- 클라우드 서비스
- 모든 제품 보기
툴
체험, 구매 & 영업
커뮤니케이션
Red Hat 소개
Red Hat은 Linux, 클라우드, 컨테이너, 쿠버네티스 등을 포함한 글로벌 엔터프라이즈 오픈소스 솔루션 공급업체입니다. Red Hat은 코어 데이터센터에서 네트워크 엣지에 이르기까지 다양한 플랫폼과 환경에서 기업의 업무 편의성을 높여 주는 강화된 기능의 솔루션을 제공합니다.