Secrets, such as usernames, passwords, API tokens, and TLS certificates, contain confidential data that can be used to authenticate and authorize users, groups, or entities. As the name implies, secrets are not meant to be known or seen by others. So how do we keep them safe?
The key to keeping secrets safe lies within how you manage them. Where to store secrets, how to retrieve them, and how to make them available in an application as needed are all early design choices a developer must make when migrating an application or microservice to Kubernetes. Part of this design choice is to ensure the secrets can become available without compromising the application’s security posture.
In this article, I will provide approaches and recommended best practices for managing secrets in Kubernetes.
How to approach secrets management in Kubernetes
Let’s start with some approaches. Below are three approaches I recommend for Kubernetes secrets management.
etcd is a supported datastore in Kubernetes, and a lot of developers opt to store secrets in a Base64-encoded format in etcd as a key-value pair. Secrets stored in etcd can be made available from within Kubernetes deployment specs as an environment variable, which is stored in memory and makes the secrets harder to extract. Secrets can also be made available from inside the container as a volume mount, but since volume mounts store secrets on a container filesystem, it is easier to extract secrets from it, compared to environment variables. If you’re worried about flexibility and security, then you’ll be happy to hear that etcd access is controlled through Kubernetes role-based access control (RBAC).
Through strong concurrency primitives, APIs, and linearizable reads, secrets can be managed at scale with etcd. However, there are drawbacks—secrets are stored, retrieved, and sent in plaintext (Base64 encoding). I strongly advise against storing secrets in plaintext because this means anyone or any machine can easily read your confidential data without you knowing. This issue can be solved by configuring etcd to encrypt data in transit or at rest using TLS.
Note that access to etcd is not audited, meaning anyone who has access to it can access all secrets. Secrets stored in etcd are also not versioned nor recoverable. For those who require large-scale organizational secrets management, etcd may not be the right choice, since it is a Kubernetes datastore.
Secrets management service
Most public cloud providers offer secrets management services. Some of the most popular secrets management services include Google Secret Manager, Azure Key Vault, and AWS Secrets Manager.
The HashiCorp Vault is another option, as it provides key management, encryption, rotation, PKI, and other essential services. As a centralized secrets manager, Vault can be used together with a cloud provider’s secrets management service. With Vault, operators can avoid handling plain-text keys, as when the Vault is initialized, initial keys can be encrypted and stored in a cloud key management service (KMS).
Container storage interface driver
The container storage interface (CSI) driver is a secret store that’s been available since
version 1.13 of Kubernetes. The CSI driver is a secret store that combines external secret stores, such as Azure, AWS, GCP, and HashiCorp’s Vault, into Kubernetes via CSI.
By authenticating your secret store service through volume attributes and mounting necessary secrets into the pod, the CSI driver approach steers clear of using the Kubernetes etcd datastore. This allows you to manage your secrets effectively and scale if needed.
Now that we’ve looked at three approaches to Kubernetes secrets management, let’s take a look at some best practices for managing secrets in Kubernetes.
Best practices for Kubernetes secrets management
Here are nine best practices for managing secrets in Kubernetes.
Avoid secrets sprawl
It’s important to avoid secrets sprawl when managing your secrets in Kubernetes. Secrets sprawl happens when application secrets are stored in different places, such as in configuration filers, Git repositories, and YAML. If there is secrets sprawl, that means no secrets management workflow is in place.
Mitigate the risk of secrets sprawl by implementing centralized secrets management. Set up a place where credentials can be securely stored and retrieved from a single point, and where they can be used by the entire organization through proper authorization, monitoring, and logging mechanisms.
Encrypt data in transit and at rest
Kubernetes does not safely store or transmit secrets by default. Hence, it is crucial to have a structure that encrypts secrets in transit with end-to-end TLS encryption, and a solution that stores secrets in an encrypted form.
Use automated secret rotation
Organizations often have varying rotation schedules for different secrets to control exposure in case of attacks. Cloud secrets management services and external solutions are usually able to provide automated secrets rotation, so your secrets can be rotated as often as hourly, depending on data sensitivity.
Have an audit log
It’s a good idea to know your secrets’ activity. In the event of a breach, an audit log provides visibility and helps you assess the incident, including whether the compromise was intentional, the affected area, and other investigative steps.
Implement anti-affinity rules
Your secrets management solution should be a single process on a limited number of designated VMs or hosts. If run as a microservice on Kubernetes, this process will be running in a dedicated pod. To specify which nodes the pods should run on, you can implement an anti-affinity rule to control the distribution of pods on the nodes that are set to run a secrets management solution.
Leverage dynamic secrets
Dynamic secrets are ephemeral secrets generated on-demand with a limited lifespan. Even if an attacker accesses a secret through means such as leaked application codes or debug logs, the secret would have been changed within a short period of time, thereby limiting exposure and protecting the application. Dynamic secrets can also make it easier to pin down the time frame a secret was discovered by an attacker and accelerate investigations.
Use a custom certificate authority
A custom certificate authority (CA) can be used to implement end-to-end TLS as part of a defense-in-depth approach. Organizations can opt to sign their certificates with their own CA, meaning the organization-signed certificate must be presented in order to access a service.
Store secrets in container memory
Storing secrets in memory will make it harder for attackers to find the secrets during a breach. Whenever a containerized application receives a secret, instead of storing the secret on a disk or in the volumeMount available in the host, it should be in memory.
Protect secret zero
Envelope encryption is often adopted by many secrets management solutions, where the data encryption keys (DEKs) are protected by a key encryption key (KEK). Here, the KEK is considered secret zero, the ultimate master key. If the KEK becomes compromised, attackers can decrypt the DEK, and consequently, the data encrypted by the DEK.
To protect secret zero, use your chosen cloud service provider’s identity and access management framework (IAM) in conjunction with key management servers (KMS). Note that both the IAM and KMS would have their own secret zero in the trust chain, which should also be considered sensitive data.
When it comes to protecting sensitive data, the most effective method is to adhere to a zero-trust model, assume compromise, and have measures in place to mitigate risks.
The approaches and best practices I have recommended in this article lay a foundation for secrets management. For those interested in having a more thorough understanding, I recommend reading chapter 3 of Kubernetes security and observability: A holistic approach to securing containers and cloud-native applications, an eBook I co-authored.
To learn more about securing containers and Kubernetes at the infrastructure level, read this free O’Reilly ebook.
Join our mailing list
Get updates on blog posts, workshops, certification programs, new releases, and more!