How to Manage Secrets in Kubernetes

Introduction

Most applications deployed through Kubernetes require access to databases, services, and other resources located externally. The easiest way to manage the login information necessary to access those resources is using Kubernetes secrets. Secrets help organize and distribute sensitive information across a cluster.

What are secrets?

A Secret is an object that contains a small amount of sensitive data such as a password, a token, or a key. Such information might otherwise be put in a Pod specification or in a container image. Using a Secret in Kubernetes means that you don’t need to include confidential data in your application code.

Because Secrets can be created independently of the Pods that use them, there is less risk of the Secret (and its data) being exposed when creating, viewing, and editing Pods. Kubernetes, and applications that run in your cluster, can also take additional precautions with Secrets, such as avoiding writing sensitive data to nonvolatile storage.

Secrets are similar to ConfigMaps but are specifically intended to hold confidential data.

Related: Kubernetes Pod Security Policy Best Practices

Why are Kubernetes secrets important?

In a distributed computing environment it is important that containerized applications remain ephemeral and do not share their resources with other pods. This is especially true in relation to PKI and other external confidential resources that pods need to access. For this reason, applications need a way to query their authentication methods externally without being held in the application itself.

Kubernetes offers a solution to this that follows the path of least privilege. Kubernetes Secrets act as separate objects that can be queried by the application Pod to provide credentials to the application for access to external resources. Secrets can only be accessed by Pods if they are explicitly part of a mounted volume or at the time when the Kubelet is pulling the image to be used for the Pod.

Learn More: Kubernetes Security: Best Practices and Tools

How does Kubernetes leverage secrets?

The Kubernetes API provides various built-in secret types for a variety of use cases found in the wild. When you create a secret, you can declare its type by leveraging the `type` field of the Secret resource, or an equivalent `kubectl` command line flag. The Secret type is used for programmatic interaction with the Secret data.

Updating and rotating secrets

Regularly updating and rotating secrets is a crucial security practice. While you can update a secret using kubectl edit, this approach is not recommended because it can be error-prone and can potentially lead to unintended consequences.

When you edit a secret with kubectl edit, you are modifying the existing secret in place. This means that any existing references to the secret (such as environment variables or volume mounts) will continue to use the old secret data until the application is restarted or the pod is deleted and recreated.

If you need to rotate a secret, you must update any references to the old secret to use the new secret instead!

That’s why past a certain size, it becomes useful to either implement app-specific mechanisms that reload configuration at runtime or deploy a sidecar container that monitors for changes and restarts the main container when necessary.

Limitations of Kubernetes secrets

While Kubernetes Secrets provide a convenient way to manage sensitive data, they have some limitations:

• Limited encryption. By default, secrets are stored unencrypted in etcd. Kubernetes does support encryption, but the encrypting key needs separate management.
• Limited rotation. Kubernetes secrets are designed to be immutable, which means that they cannot be modified or versioned once they are created. This, as we have seen, makes it difficult to rotate secrets. They can’t be audited either.
• Limited access control. While Kubernetes provides RBAC (Role-Based Access Control) to control access to secrets, it is still possible for unauthorized users to gain access to secrets if they can compromise the cluster or the underlying infrastructure.
• They may not be suitable for large-scale or highly regulated environments, where more advanced secret management solutions might be necessary.

Despite these limitations, Kubernetes Secrets remain a handy tool for managing secrets when you don’t need to scale immediately your cluster.

Kubernetes external secrets

Kubernetes External Secrets offer an alternative approach to managing secrets in Kubernetes by integrating with external secret management solutions. This allows you to maintain sensitive data outside of your Kubernetes cluster while still providing seamless access to applications running within the cluster.

How does it work?

Kubernetes External Secrets are custom resources that act as a bridge between your Kubernetes cluster and external secret management systems.

Instead of storing secrets directly in Kubernetes, External Secrets fetch and synchronize secrets from external systems, making them available as native Kubernetes Secrets. This ensures that your applications can access sensitive data without any code changes while benefiting from the security features provided by external secret management solutions.

Integrating with external secret management solutions

Kubernetes External Secrets can integrate with a variety of external secret management solutions, such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.

To integrate External Secrets with your chosen secret management system, you need to deploy the corresponding External Secrets controller in your cluster and configure it to communicate with the external system.

For example, to integrate with HashiCorp Vault, you would deploy the Kubernetes External Secrets controller for Vault and configure it with the necessary Vault authentication and connection details.

Creating and using external secrets in Kubernetes

To create an External Secret, you need to define a custom resource in a YAML file, specifying the reference to the secret stored in the external system:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: vault
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/my-secret
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/my-secret
        property: password

Apply the YAML file using kubectl apply -f my-external-secret.yaml

The External Secrets controller will fetch the secret data from the external system and create a native Kubernetes Secret with the same name. This generated secret can be used by your applications in the same way as regular Kubernetes Secrets.

Advantages of Kubernetes external secrets

Using Kubernetes External Secrets offers several benefits:

• Enhanced security by leveraging the features of external secret management solutions, such as encryption, access control, and auditing.
• Reduced risk of exposing sensitive data within the Kubernetes cluster.
• Simplified secret management for organizations already using external secret management systems.
• Centralized secret management across multiple Kubernetes clusters and other platforms.

By integrating Kubernetes External Secrets with your chosen secret management solution, you can achieve a higher level of security and control over your sensitive data while maintaining compatibility with your existing Kubernetes applications.

Best practices for managing secrets in Kubernetes

To ensure the security and integrity of your sensitive data, it is crucial to follow best practices for secret management in Kubernetes. Below are some of the most important practices to keep your secrets secure and maintain a robust Kubernetes environment.

Role-based access control (RBAC)

RBAC is essential for managing secrets securely, as it enables you to control which users and components can create, read, update, or delete secrets. By implementing fine-grained access control, you can minimize the risk of unauthorized access and potential data breaches.

To implement RBAC for secrets management, you should create roles and role bindings that define the allowed actions on secrets for each user or group. For example, you can create a role that allows read-only access to secrets within a specific namespace and bind it to a specific user or group:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Encrypting secrets at rest and in transit

Encrypting secrets is crucial for protecting sensitive data from unauthorized access, both when stored in etcd (at rest) and when transmitted within the cluster (in transit).

Kubernetes provides native encryption options, such as enabling etcd encryption to protect secrets at rest and using TLS for securing communications within the cluster. Ensure these options are configured and enabled to maintain the confidentiality of your secrets.

In addition to Kubernetes native encryption options, you can also integrate third-party encryption solutions, such as HashiCorp Vault or cloud-based key management services, to further enhance the security of your secrets.

Secret rotation and expiration

Regularly rotating secrets is an essential security practice that minimizes the risk of unauthorized access and potential data breaches.

Strategies for secret rotation include manual updates using kubectl or automated rotation using custom controllers or third-party secret management solutions.

Automating secret rotation can be achieved using Kubernetes operators, external secret management systems, or custom scripts that periodically update secrets based on a predefined schedule or events.

Auditing and monitoring

Auditing and monitoring are crucial for maintaining the security and integrity of your secrets, as they enable you to track and analyze secret access, usage, and modifications and detect potential security incidents.

Several tools can be used for auditing and monitoring secrets, such as Kubernetes audit logs, Prometheus, Grafana, and commercial solutions such as Mend.io that provide detection against hard-coded secrets.

Configure alerts and notifications to proactively notify administrators of potential security incidents or irregular secret access patterns, enabling timely investigation and response to potential threats.

Final thoughts

In this blog post, we have discussed the importance of secrets management in Kubernetes and explored various methods and best practices for securely managing sensitive data in your applications. Kubernetes Secrets and Kubernetes External Secrets provide powerful tools and methodologies for managing secrets effectively, ensuring the security and integrity of your sensitive information.

Managing secrets securely in Kubernetes is critical for maintaining the confidentiality and integrity of your sensitive data. By following best practices and methodologies, you can create a robust and secure Kubernetes environment, safeguarding your applications and data from potential threats.