Image
How to encrypt etcd and use secrets in OpenShift
Secrets contain sensitive information, but they aren't encrypted by default. Learn how to encrypt an etcd database to manage vital information in OpenShift.
A secret in OpenShift is any critical resource or object type that provides a mechanism to hold sensitive information, such as a password, sensitive configuration file, Transport Layer Security (TLS) certificate, Secure Shell (SSH) credential, or OAuth token.
A secret is used in internal connections between different resources. For example, applications and services can share secret data within a namespace.
Types of secrets
There are different types of secrets, and OpenShift validates that data stored in a secret conforms to the type of secret in use. Secret types include:
- kubernetes.io/service-account-token: ServiceAccount token
- kubernetes.io/basic-auth: Credentials for basic authentication
- kubernetes.io/ssh-auth: Credentials for SSH authentication
- kubernetes.io/tls: Data for a TLS client or server
- Opaque: Arbitrary user-defined data
Most secret types you create will likely be of the opaque type. The opaque secret type is used in cases where you don't want validation, meaning that the secret does not claim to conform to any conventions for key names or values.
OpenShift uses secrets for two primary reasons:
- To store credentials used by pods in a microservices architecture. A secret is useful when you want to connect two pods.
- To store TLS certificates and keys. Secrets are crucial in defining secure routes in OpenShift networking. Developers can mount a secret as a volume and create a pass-through route to an application. It's important to note that a TLS secret stores the certificate as
tls.crt
and the certificate key astls.key
, so the application using them must use the same naming convention.
[ Check out this guide to boosting hybrid cloud security and protecting your business. ]
Create a secret object on the controller node
You must specify the secret type (for example, generic, TLS, or docker-registry) when you create it.
Use the --from-literal
flag to create a generic secret:
$ oc create secret generic topsecret \
--from-literal \
user=vcirrus-consulting \
--from-literal \
password=topsecretpassword \
secret/topsecret created
The secret looks like this after you create it:
$ oc get secret topsecret -o yaml
---
apiVersion: v1
data:
password: dG9wc2VjcmV0cGFzc3dvcmQ=
user: dmNpcnJ1cy1jb25zdWx0aW5n
kind: Secret
...
It's important to know that a secret is not encrypted. It's just a ConfigMap encoded with Base64, so anyone can decode it into plain text:
$ echo dmNpcnJ1cy1jb25zdWx0aW5n | base64 --decode
vcirrus-consulting
$ echo dG9wc2VjcmV0cGFzc3dvcmQ= | base64 --decode
topsecretpassword
To protect data in a secret, you can encrypt etcd. Etcd is a Kubernetes data store that contains cluster information in key-value pairs.
[ Learn Kubernetes usage basics in this cheat sheet. ]
Create a generic secret from a file
Use the --from-file
option to create a generic secret using the contents of a file:
$ oc create secret generic top-ssh-secret \
--from-file secure_id_ecdsa \
--from-file secure_id_ecdsa.pub
secret/top-ssh-secret created
The secure_id_ecdsa
and secure_id_ecdsa.pub
files contain the private and public SSH keys, respectively.
Expose secrets to pods
To make a secret available to a pod, you can refer to a secret as a variable or as a file in the pod's configuration. The most convenient way to update a pod with information in a secret is by using the oc set env
command. This writes the environment variables obtained from a secret to a pod or deployment.
1. Create the secret
First, create a generic secret with the variables you want to be used within the pod:
$ oc create secret generic \
mysql-secret --from-literal user=janedoe \
--from-literal password=mysqlpassword \
--from-literal database=mysqlsecretdb \
--from-literal hostname=janedoe-mysql \
--from-literal root_password=janedoe-password
secret/mysql-secret created
2. Use jq to view the secret
Use the jq
command to view the secret you've created as JSON:
$ oc get secret mysql-secret -o json | jq
{
"apiVersion": "v1",
"data": {
"database": "bXlzcWxzZWNyZXRkYg==",
"hostname": "amFuZWRvZS1teXNxbA==",
"password": "bXlzcWxwYXNzd29yZA==",
"root_password": "amFuZWRvZS1wYXNzd29yZA==",
"user": "amFuZWRvZQ=="
},
"kind": "Secret",
"metadata": {
"creationTimestamp": "2022-08-22T11:59:41Z",
"name": "mysql-secret",
"namespace": "default",
"resourceVersion": "163733",
"uid": "b6f43859-ad60-49dd-9be8-3d23f68cd5de"
},
"type": "Opaque"
}
[ Want to test your sysadmin skills? Take a skills assessment today. ]
3. Start a MySQL pod
Now start a MySQL or MariaDB pod with the name janedoe-mysql
:
$ oc new-app --name janedoe-mysql --image bitnami/mysql
[…]
--> Success
Application is not exposed. You can expose services to the outside world by executing one or more of the commands below:
'oc expose service/janedoe-mysql'
Run 'oc status' to view your app.
4. Watch the pod fail
Watch the status of the pod. Note that the pod fails with the CrashLoopBackOff error:
$ oc get pods -w
NAME READY STATUS RESTARTS AGE
janedoe-mysql-d5dd--- 0/1 ContainerCreating 0 8s
janedoe-mysql-d5dd--- 0/1 Error 0 2m44s
janedoe-mysql-d5dd--- 0/1 Error 1 (2s ago) 2m45s
janedoe-mysql-d5dd--- 0/1 CrashLoopBackOff 1 (2s ago) 2m46s
5. Investigate
The pod failed, and you can learn why by viewing the output of oc logs
. In this case, the Bitnami MySQL pod is failing because an environment variable isn't set. If you're running a MariaDB pod, oc logs
hints at which variable you need to set:
$ oc logs janedoe-mysql-d5ddd6877-nqntj
mysql 12:07:49.52
mysql 12:07:49.52 Welcome to the Bitnami mysql container
mysql 12:07:49.53 Subscribe to project updates by watching https://github.com/bitnami/containers
mysql 12:07:49.53 Submit issues and feature requests at https://github.com/bitnami/containers/issues
mysql 12:07:49.54
mysql 12:07:49.54 INFO ==> ** Starting MySQL setup **
mysql 12:07:49.59 INFO ==> Validating settings in MYSQL_*/MARIADB_* env vars
mysql 12:07:49.60 ERROR ==> The MYSQL_ROOT_PASSWORD environment variable is empty or not set. Set the environment variable ALLOW_EMPTY_PASSWORD=yes to allow the container to be started with blank passwords. This is recommended only for development.
6. Create a secret
To satisfy the pod's requirement, you must set a root_password variable in a secret. Refer to Step 1 above to create a new secret containing the root_password variable, but include the prefix MYSQL_.
When you include the ROOT_PASSWORD variable and exclude the MYSQL_ prefix, you make your secret flexible enough to work in multiple environments. In this case, the secret could adapt well, for example, to either a MySQL or a MariaDB environment.
Using a prefix is convenient because it allows you to define a variable and use it in a specific way in your configuration. This example defined root_password, which is not significant in a MySQL environment. MySQL requires a variable with the name MYSQL_ROOT_PASSWORD.
7. Update the failing pod with a secret
The Bitnami MySQL image requires the MYSQL_ROOT_PASSWORD variable. The command below reads variables from your secret, and it can detect the correct variable because one of them is root_password combined with the MYSQL_ prefix:
$ oc set env deployment/janedoe-mysql \
--from secret/mysql-secret --prefix MYSQL_
deployment.apps/janedoe-mysql updated
8. Watch the pod again and confirm it's in a running state
Now that you've updated your secret, watch the status of the test MySQL pod and verify that it's running:
$ oc get pods -w
NAME READY STATUS RESTARTS AGE
janedoe-mysql-7c56--- 1/1 Running 0 11s
[ Getting started with containers? Check out this no-cost course on deploying containerized applications. ]
9. Explore the pod environment
Verify the environment variables provided by your secret by listing all of them with the env
command:
$ oc exec -it \
janedoe-mysql-7c567d5564-d99pp – env
[...]
MYSQL_HOSTNAME=janedoe-mysql
MYSQL_PASSWORD=mysqlpassword
MYSQL_ROOT_PASSWORD=janedoe-password
MYSQL_USER=janedoe
MYSQL_DATABASE=mysqlsecretdb
KUBERNETES_PORT_443_TCP=tcp://10.217.4.1:443
JANEDOE_MYSQL_PORT_3306_TCP_ADDR=10.217.5.245
KUBERNETES_PORT=tcp://10.217.4.1:443
JANEDOE_MYSQL_PORT_3306_TCP_PORT=3306
KUBERNETES_SERVICE_PORT_HTTPS=443
[...]
Encapsulate your data
Despite its name, a secret isn't secret until you encrypt etcd, but it's still a useful way to pass vital information into pods.
Now that you understand how to use a secret, try learning how to mount a secret as a volume (hint: try the oc set volume
command).
Keep in mind that you should keep your secrets private on production servers. Read Encrypting etcd data in the OpenShift docs for more information.
Image
Learn how to configure service account access restrictions and security context constraints (SCCs) to control permissions for pods.
Image
Learn how to change a default security context constraint (SCC) in OpenShift to manage permissions within a cluster.
Image
Learn how containers communicate within a pod through the same Kubernetes network namespace
Robert Kimani
Robert is a Linux enthusiast and an open source advocate, currently transitioning into a site reliability engineering (SRE) role. Always striving to learn more, he's pursuing Red Hat Certified Architect - Infrastructure path certification. Besides his love for Linux, he believes in helping others More about me