Encrypting Secret Data at Rest

Introduction

A "secrets" object was added to Kubernetes v1.3 to provide a way to secure user credentials and other sensitive data. This data could be accessed at runtime for service authentication and other purposes. Building this data directly into images or passing it as environmental variables could compromise overall security.

Storing secret data in etcd was the initial solution. However, this initial design had some risks associated with it and several guidelines were issued to handle them: Restrict access to etcd to admin users, applications should exercise control over how secret data was being stored and transmitted across third party applications, and so forth. But implementing the recommendations in these guidelines do not automatically prevent organizations from accidentally exposing secrets.

Additional Security

To improve the security practices concerning secrets, encryption of secrets at rest will be introduced in the upcoming OpenShift 3.6.1 release. With this release, users will have the option to encrypt data before storing it to etcd. This adds an additional layer of security on top of:

• base64 encoded secret data
• secrets protected in transit via TLS
• full disk encryption of the volumes storing etcd data

The new encryption defends against malicious parties gaining access to data stored in ectd.

Configuring Encryption

In order to activate support for data encryption, an argument --experimental-encryption-provider-config should be passed to the Kubernetes API server:

Excerpt of master-config.yaml

kubernetesMasterConfig:

apiServerArguments:

experimental-encryption-provider-config:

- /path/to/encryption.config

After editing the file, restart OpenShift to enable secrets encryption.

Encrypting Data

Currently, three types of encryption providers are supported: aescbc, secretbox, and aesgcm. aescb is the recommended choice for encryption at rest. Each provider supports multiple keys and the keys are tried in order for decryption. If the provider is the first entry in the config file, the first key is used for encryption. If you want to encrypt/decrypt data, take a look at this document for more information.

Rotating Encryption Key

The encryption keys are stored on disk on all the api-server nodes. If aesgcmencryption is used, keys should be rotated every 2^{^}32 writes.

Conclusion

When implementation and usage deviates from the API usage best practices, resulting in privilege escalation issues, this new feature for encrypting data at rest provides improved security.