Kubernetes Security Best Practices

Kubernetes is an open source orchestration platform for containerized workflows. It is the best way to manage – or orchestrate – large clusters of containers at scale. Sometimes abbreviated as K8s, Kubernetes helps you efficiently manage clusters of hosts running Linux containers.

In the age of containers, Kubernetes has become a popular open source project and key building block for modern tech infrastructure. Containers need to be managed, and Kubernetes helps with tasks such as scheduling, load balancing, and distribution, while making sure container workloads run as intended. With Kubernetes, you can deploy and manage cloud-native applications anywhere independent of cloud-specific requirements, avoiding infrastructure lock-in.

Kubernetes, which is Greek for helmsman or pilot, is exploding in popularity. According to a survey of the Cloud Native Computing Foundation (CNCF) end user community, Kubernetes use has increased from 58% of organizations using or evaluating the technology in 2018 to 96% in 2022. With their rise in popularity, security should be a primary concern and not an afterthought.

What is Kubernetes security?

Kubernetes security is a set of tools, processes, and practices that are recommended to keep your Kubernetes platform secure. Securing Kubernetes platforms can be quite complicated, because Kubernetes itself is quite a complex system of components. So, it’s important to understand how it works and how various kinds of vulnerabilities can enter your system when using Kubernetes.

Making Kubernetes secure involves addressing these different vulnerabilities and risks that can affect the different layers of a Kubernetes cluster. To take a comprehensive approach, you need to ensure that you implement security in application clusters, containers, code, and microservices.  

Kubernetes security and the cloud-native environment

Though Kubernetes has several built-in security advantages -– for example, container images are usually replaced, not patched or updated, which leads to better version control -– its increase in use means more opportunity for exposure. Because Kubernetes is a relatively new and complex technology, it is also highly attractive to hackers.

When addressing Kubernetes security, it is nearly impossible to separate the Kubernetes cluster from the other layers of cloud computing. These layers include the application or code itself, the container image, the Kubernetes cluster, and the cloud or networking layer as shown in the image below. Each layer builds on the previous layer, and all layers must be secured to achieve defense in depth.

Kubernetes security issues and DevOps

Another way to think more specifically about Kubernetes security from a developer point of view is to look at it across three distinct stages of the DevOps pipeline:

• Creating the container image during the build
• Configuring the Kubernetes infrastructure during deployment
• Securing network communication in production

Kubernetes security during build

Kubernetes security begins in the build stage when writing the code that becomes the container image. Although Kubernetes isn’t necessarily involved during the build and packaging stage, a secure environment starts with assessing the risk associated with any code being deployed. Application security best practices and testing are important here, and any effort to shift security left will pay dividends by avoiding future problems in deployment and production.

In this stage, you need to make sure that your container image is up to date, free from security vulnerabilities, and is compliant with your licensing policies. Containers are built-in layers, and often have no canonical list of what open source components have been included in the final build of the container image. To address this, you need to scan containers to determine which open source libraries are in play inside the application, including going through any dependencies that are brought in through a package manager like Maven or npm. By discovering and identifying all open source components in play, you can identify any known vulnerabilities, establish which images are at risk as new vulnerabilities are discovered, and track the licenses associated with specific libraries. Tracking license compliance may not be a security risk, but it can have a downstream effect like exposing proprietary code if an incompatible license is included.

Ensuring your base images are regularly scanned is the first step to securing your Kubernetes environment. 

Kubernetes security during deployment 

When deploying Kubernetes, you are given a powerful set of controls to secure clusters and their applications. Configuring these controls requires an intimate knowledge of Kubernetes as well as the deployment’s security requirements. Don’t use defaults! It only leads to unnecessary (and perhaps even negligent) exposure. At the very least, you should limit the scope of user permissions, limit direct access to Kubernetes nodes, and create network segmentation so that containers communicate only with the containers they are supposed to.

Scanning at this stage remains of paramount importance. Container images must be built using secure base images. You need to implement continuous security vulnerability scanning and regularly apply security updates to your environment. Scanning is not a one-off event. In addition, when launching containers in your Kubernetes environment, only use images from image registries on allow lists.

Kubernetes security in production

Application security in runtime is very different from the build stage. Notably, network security is more complex. Best practices for network security in Kubernetes go beyond basic networking and leverage the container network interface (CNI) to implement a more robust networking layer that includes either multi-tenant support, network policies, or both.

With a multi-tenant network, each namespace within the Kubernetes cluster gets a private addressable subnet and can only access pods in other namespaces that are specifically exposed as services. Open vSwitch is a great example of a networking layer for Kubernetes that has multi-tenant support.

Network policies in most next-generation networking layers are available to be deployed via Kubernetes. Project Calico is an example of a widely used networking layer that has been built from the ground up to support network policies with containers. Network policies allow cluster administrators to predefine network access control lists that control which ports are accessible and which services can be accessed.

Kubernetes security best practices

While this is far from an exhaustive list, here are some best practices for Kubernetes security at various stages to get you started.

• Scan your image and source code. As with any application, implementing application security testing best practices of using various scanning tools such as SAST, DAST, IAST, or SCA will help ensure your code is as secure as possible.
• Don’t forget about open source. Open source code is in almost every proprietary software offering on the market, making up approximately 60%-80% of all software codebases. If you’re not actively managing your open source code, you’re not secure.
• Implement workflows to remediate vulnerabilities. Automate your workflows to ensure that vulnerabilities are remediated as soon as possible and code is secure.
• Use updated images. This one goes without saying, but make sure you’re using the most updated image available as it will likely be the most secure image available. Destroy any outdated images; it’s not really possible to patch them.

Kubernetes security during deployment

• Don’t deploy images from unknown sources. If you don’t know the provenance of an image, you can’t trust it.
• Scan images during deployment. You might find new vulnerabilities that have been disclosed since your image was last scanned in development.
• Use image admission controls. Admission controls stop an image if it violates your org’s security policies and prevent a container from running with greater privileges than it needs for its tasks.
• Limit privileges used by a container. Giving a container broad permissions can be problematic if a malicious user gets control of it. Limit your exposure by limiting a container’s privileges and never run a container’s privilege flag.
• Don’t use defaults. Enable role-based access control (RBAC) in your container orchestration to regulate access based on the roles of individual users within your organization.

Kubernetes security in production

• Scan images in production. Again, we can’t stress this enough: new vulnerabilities are disclosed all the time. Make sure you’re meticulous about continuously scanning your images.
• Secure the network with a container firewall. You can secure and inspect your network using a container firewall, which applies traditional network security techniques to the cloud-native Kubernetes environment.
• Implement network security policies. Keep an eye on your network traffic flows by creating and defining cluster network policies that allow you to control network access in and out of your containerized applications.
• Establish security boundaries using namespaces. Namespaces are virtual clusters in Kubernetes. They allow you to divide cluster resources between multiple users.
• Create a security policy for pods. Pods, the smallest execution unit in Kubernetes, are used to organize containers. Pod security policies can be used to address issues such as a container running as root.
• Analyze pods in the same deployment. When replicated, pods should exhibit identical behavior. Monitor and analyze pods. If you see them behaving differently, you need to investigate immediately to ensure no breach has occurred.

Securing Kubernetes

• Update Kubernetes frequently. Kubernetes support covers only the most three recent versions. If you’re not updating Kubernetes regularly, you will miss out on patches for newly disclosed security vulnerabilities. As a bonus incentive, new security features are added in every quarterly Kubernetes update. Don’t expose your organization because of a basic oversight.
• Kubernetes API. Do not leave the Kubernetes API publicly exposed and only allow access via an internal network or VPN.

Kubernetes security context

Kubernetes security context defines privileges and access control settings for pods and containers. The security context establishes the security parameters that are assigned to each pod or container. Think of it as a set of rules for determining how and by whom a given resource should be accessed. And remember: don’t run as root!

The following table shows some of the most important Kubernetes security context parameters.

Kubernetes security tools

One of the challenges with Kubernetes and containers is a lack of visibility, which makes it harder to remediate vulnerabilities. Because images are continuously spun up and taken down, scans are more difficult to complete as the system is in constant flux. 

When looking for Kubernetes and containers security tools, look for tools that were built to handle the range of threats specific to Kubernetes. Prioritize tools and vendors that offer native support with your chosen Kubernetes distribution. Also look for tools that scan pods deployed to production for vulnerabilities or compliance issues and that enforce policies automatically.

What to secure, and how

Hardening your Kubernetes security and ensuring that it’s comprehensive, requires you to know how to secure Kubernetes nodes, pods, networks, and data. Below is a summary of the main elements you need to secure, how to secure them, and what you need to use to do so.

Nodes house Kubernetes clusters, usually in Linux but sometimes in Windows. To secure them you should take steps to reduce your attack surface by removing irrelevant, unnecessary, and unused applications, libraries, other components, and user accounts. Also, deploy operating system hardening and analyze OS logs to detect potential breaches. These strategies are similar to those used to secure servers.

Pods are containers that run your applications. You should perform security tests on any applications prior to deployment and scan container images before you run them. And you should collect and analyze logs from pods to detect any potential breaches or malicious behavior. Also, use Kubernetes’ native tools to strengthen security, such as role-based access control (RBAC) policies and security contexts, to limit and police access to pods; network policies (see below); and admission controllers that provide an extra layer of defense against unauthorized requests for access.

Networks also benefit from taking similar measures to those you implement for pod security. Use external tools like a service mesh to secure networking configurations outside of Kubernetes and monitor network traffic to check for breaches. Additionally, use Kubernetes’ native network policies to help control traffic flows within clusters. Taking these steps enables you to isolate pods to prevent infection and filter incoming traffic.

Data

Kubernetes doesn’t store any data, except for data that’s inside running pods and log data stored on nodes. The data that you access with Kubernetes is stored elsewhere. So, you need to take the typical steps necessary to secure data in other systems. Apply encryption, access restrictions, and back up all data.

Additional security measures that you’re advised to take are:

• Implement Kubernetes API authentication and authorization, governed by RBAC policies that assign permissions, privileges, and access
• Use admission controllers
• Configure secure certificates and requiring the API server to serve requests on a secure port rather than on a local host
• Establish an audit policy, and keep detailed audit logs of who performed what actions within a cluster, so you can identify suspicious incidents
• Isolate different workloads in separate namespaces to reduce the opportunity for infection or breaches between each of them

Kubernetes security audit

Organizations are continuously updating and upgrading the Kubernetes environment as the technology matures. Periodic audits to determine the health and security of your Kubernetes deployment are recommended. When completing an audit, focus on the following areas:

• Host security
• Kubernetes security
• Docker daemon security
• Container security
• Properly configured role-based access controls (RBACs)
• Securing data at rest and in transit
• Vulnerability scanning of images and containers

A comprehensive Kubernetes security audit helps you prevent known exploits.

Reducing Kubernetes attack surfaces

In their complexity, cloud-native environments like Kubernetes present new and evolving challenges when implementing security best practices and meeting internal security policy and compliance goals. Using the tips laid out here as a starting point, you can begin to harden your Kubernetes environment and reduce attack surfaces. Through effective security best practices, you can gain both visibility and control over every layer of your Kubernetes deployment.

FAQs

Is Kubernetes a security risk?

Kubernetes, like any other technology, has its own set of security risks. One of the biggest security risks is the possibility of a malicious actor gaining access to the Kubernetes control plane, which would allow them to control all of the resources in the cluster. Additionally, Kubernetes pods are by default able to communicate with each other, which can lead to privilege escalation and data breaches if not properly secured.

Kubernetes also requires proper configuration to secure communication between nodes and pods. If not configured properly, it can lead to a data breach.

How do you maintain security in Kubernetes?

Maintaining security in a Kubernetes environment requires a combination of good practices, processes, and tools. One of the most important steps is to properly configure access to the Kubernetes control plane. This can be done by using role-based access control (RBAC) and network segmentation to limit access to only authorized users and systems.

Another important step is to properly configure pod security policies to limit the level of access and privileges that pods have within the cluster. This can be done by using network policies and namespaces to limit communication between pods.

Additionally, it is important to use security-conscious practices when deploying containers, such as properly configuring resource limits, using security-hardened images, and regularly patching and updating the cluster.

Another way to maintain security in Kubernetes is to use Kubernetes security tools like Network Policies, Pod Security Admission, and Service Accounts for Pods to control the network traffic, pod access and permissions, and audit.

What is the biggest disadvantage of Kubernetes?

The biggest disadvantage of Kubernetes is its complexity. Kubernetes has a steep learning curve and requires a certain level of expertise to properly configure and manage. Additionally, it can be difficult to troubleshoot and debug issues that arise within a Kubernetes cluster.

Secure containerized applications

See Mend Container