4 Kubernetes Secrets Management Best Practices

Originally published by New Context.
 

Most people can probably remember being reproved for whispering as a child. The reason given for this sometimes-embarrassing admonishment ranged from the whispering being considered offensive to it excluding others who were being left out of the conversation. While these are important lessons in a child’s life, when it comes to data security, almost everyone agrees that keeping sensitive information secret far outweighs any offense an excluded party may experience.

In today’s highly competitive software development and operations (DevOps) landscape, security should be a driving factor and not functionally or strategically separate. This issue is particularly important when a tool such as Kubernetes is utilized. Kubernetes does come with inherent security capabilities; however, the best security may also be achieved by integrating Kubernetes secrets management with a third-party tool.

Kubernetes Secrets?

For enterprises, sensitive information can be difficult to broadly define. For some industries, such as banking or healthcare, virtually all transactions or data exchanges may involve privileged information that could be used to identify users or patients. For other companies, the restrictions may not need to be as extensive.

Regardless of industry, however, any credentials that are used to allow specific access are indeed sensitive and therefore require security restrictions. These secrets include passwords, security keys, digital certificates, tokens, and other user authentication identifiers that must remain unknown to everyone except a finite number of authorized persons or services.

Kubernetes is an open-source workload orchestration and runtime platform that inherently provides many infrastructure components and deployment conventions, including secrets management. The included secrets management capability allows application-specific secrets to be configured in addition to those that are automatically created for service accounts. Moreover, it is not necessary to have access to specific application code to utilize secrets as they can be attached to an application workload via native Kubernetes functions (i.e., API call).

Although Kubernetes Secrets is an option to be considered for handling sensitive information in production environments, there are potential trade-offs that require attention. For example, the default state of secrets being unencrypted at rest (within etcd) could potentially expose them via unchecked generic administrator access. These deficiencies (if not properly compensated for) could lead to not only compromised security credentials, but also costly breaches of corporate data and personally identifiable information (PII), or possibly even the stability of system operations. In light of these challenges, let’s take a look at what best practices to apply when working with Kubernetes.

‍

Secrets Management when Using Kubernetes

To effectively protect your organization’s operations requires the incorporation of security throughout the development process. An essential aspect of this strategy is securing the identity and access credentials utilized for your applications. There are many threats to these identity credentials. For example, certificates can expire, keys can be lost, and passwords can be hacked. By following the suggested practices below, you can make the best use of Kubernetes, while providing access to secrets when needed and restricting unauthorized usage, thus minimizing or completely avoiding obstacles to smooth and secure cloud data management.

1. Pod Creation Best Practices

During development, it’s not uncommon for developers to own functionality beyond application features and therefore create code for credential utilization within their application. However, this enablement, left unchecked, could introduce poor credential handling practices, such as deploying from and/or checking in deployment manifests that directly contain raw Secret objects. To alleviate these risks, consider implementing a separate security workflow to handle secrets configuration as distinct from application configuration, and then leverage a reference pattern (corresponding secret by name, id, or other object) within the application manifest. This enables more stringent control over how and when actual sensitive secrets data is managed.

2. Configuration Best Practices

Almost all data within the Kubernetes API is stored within the distributed data store, etc, which includes Secrets. Ensuring that the Kubernetes cluster configuration is set to encrypt all data at rest (within etcd) is critical to reducing the risk of broad compromise. Leveraging the cluster-level EncryptionConfiguration object enables this encryption at rest and reduces the number of keys that need to be properly managed to a mere handful. Ideally, managing these cluster keys with an external provider (such as a cloud provider’s KMS service) is the best route as it moves the “first key problem” outside of Kubernetes itself.

3. Networking Best Practices

Applying the least-privilege concept to network traffic can provide extra assurances that sensitive Kubernetes cluster components (such as the API server and etcd) are only accessible from the services that need direct access. Consider locking down service-to-service traffic to only the required ports and sources via your cloud provider’s traffic filtering mechanisms (e.g., AWS Security Groups). Also, taking this a step beyond cluster component traffic, restricting traffic between application components should also be considered and could be achieved by leveraging a NetworkPolicy tool (e.g., Calico).

4. Security Best Practices

By default, Kubernetes administrators can access all Secrets stored within the cluster. Infrastructure/cloud administrators can potentially access the node’s storage and/or backups, which may contain unencrypted data (unless the steps in #2 were taken to encrypt data at rest). Therefore, restricting these privileges is critical to avoid unintended access of secrets. Leveraging both Kubernetes RBAC and your cloud provider’s RBAC (if applicable) is crucial to maintaining proper access for different staff, as well as audit and logging capabilities for incidents where a trusted insider may be involved.

Following these best practices will result in more robust Kubernetes security management. However, replacing or supplementing the above patterns with a third-party tool for secrets management may be the best fit for your organization.

Alternative Kubernetes Secrets Practices

The security issues inherent with Kubernetes secrets management has led to the development of alternative approaches that allow for more flexible and robust security. One of these is Sealed Secrets, which is an open-source Kubernetes controller that enables Secrets data to be safely stored in git and can be customized to address specific access issues. Another option is to use the popular HashiCorp Vault tool. Vault is a well-tested and dependable option that includes dynamic secrets generation, replication across region, cloud, and datacenter, and can be used with various authentication methods.

Applying Kubernetes Secrets Management Best Practices

As outlined above, Kubernetes secrets are potentially accessible by unauthorized users, especially if the precautions listed above are not followed. Therefore, the best practice may be to opt for a less complex, more powerful alternative as discussed above. In either case, the best implementation option is to enlist an experienced DevSecOps partner who is fully committed to providing you with the most secure environment.