Kubernetes is growing in popularity, and its use within organizations is maturing. According to the latest Cloud Native Computing Foundation (CNCF) annual survey, 93% of organizations are using or planning to use containers in production, and 96% are using or evaluating Kubernetes. According to a previous survey, 28% of organizations have more than 11 Kubernetes production clusters.
At the same time, security concerns around Kubernetes are mounting. A Red Hat survey of Kubernetes adoption and security showed that, of 500 DevOps professionals surveyed:
Read on to better understand the security risks facing Kubernetes environments, understand what security mechanisms Kubernetes natively provides, and discover best practices for improving your Kubernetes security posture.
This is part of an extensive series of guides about Kubernetes.
In this article, you will learn:
Below are critical best practices you must implement to secure your Kubernetes clusters, divided into the three main stages of the development lifecycle: build, deploy, and runtime.
Here are key best practices that will help you secure containers during the build phase of your software development lifecycle (SDLC).
It is essential to make sure that container images are free of vulnerabilities, because every container created from an image will inherit its vulnerabilities. This is typically done by scanning the base image and all packages against a vulnerable database.
Image-scanning tools verify that:
Images must be scanned at all stages of the CI/CD pipeline, and access to image registries must be controlled to avoid tampering.
Ensure that containers only have the minimal required privileges on the host. Use a hardened host operating system with adequate controls to restrict system calls and file system access, and have strong isolation between processes. This can help prevent privilege escalation attacks, where a compromised container gains access to the host operating system, or other containers running on the system.
When selecting a base image for your containers, use an image with the minimal software packages absolutely necessary for your container to function. It is even better to build a container without a full base image:
FROM scratch directive to create a minimal image and explicitly select the packages you want to add to it.Learn more in our blog post: What Your Kubernetes Security Checklist Might Be Missing
When the time comes to deploy your containers to production, you can secure them using the following measures.
Kubernetes cluster configuration is not secure by default. To ensure your clusters are secure, follow these steps:
Typically, existing security tools like web access control gateways and Next Generation Firewalls (NGFW) are not aware of Kubernetes resources. You can overcome this by integrating your Kubernetes clusters with the existing security toolset. This allows perimeter security systems to work together with security measures deployed inside your clusters.
An effective way to do this is to feed the IP addresses and TCP/UDP ports used by your workloads into perimeter security tools, on an ongoing basis, to allow them to identify and secure Kubernetes assets.
Another approach is to use cloud provider security groups to limit network connectivity to and from Kubernetes nodes. Security groups are more aligned with Kubernetes architecture than traditional security tools.
Here are best practices that can help you secure Kubernetes cluster infrastructure and workloads during runtime.
A critical concern in Kubernetes environments is that security controls in corporate networks, as well as security groups managed by cloud providers, are all focused on the node level. You can control traffic to and from a node, but existing controls do not know which pod or service is running on a node. This makes security controls largely ineffective at runtime, because different services may be running on the same node at different times, each with its own security requirements.
Kubernetes workloads are dynamic, and automated CI/CD processes constantly deploy different services, or new versions of the same services, on Kubernetes cluster nodes. To further complicate matters, the same workloads can transition between on-premises and different cloud environments, each of which has its own network security controls.
To achieve network security in a Kubernetes environment, you must build network security definitions into your workloads, using a declarative model. Security definitions must be an inseparable part of Kubernetes workloads, and must be portable across Kubernetes distributions and data centers. Wherever the workload runs, it must always carry its security definitions with it. This can be achieved in two ways:
Related content: Read our guide to Kubernetes network policy
In addition to network security controls, apply the following enterprise security controls to your Kubernetes environment:
Related content: Read our guide to securing Kubernetes host endpoints
Malicious activity in a Kubernetes cluster can involve threat actors exploiting vulnerabilities in components running in containers, exploiting insecure configurations, or directing malicious traffic at nodes, pods, or the Kubernetes control plane.
To defend a Kubernetes cluster against threats, you need two key capabilities:
A key challenge when attempting to detect intrusions is that a Kubernetes cluster generates huge amounts of log data. Use the following guidelines to make data more manageable, and derive security insights.
Related content: Kubernetes security policy design best practices
Traditional security tools cannot handle a highly dynamic environment like Kubernetes, which constantly deploys new containers and retires old ones, especially at a large scale. For example, using a single peripheral firewall for the entire application can expose you to greater risk, because once threat actors breach the firewall, they gain access to the entire system.
In addition to the dynamic nature of containers, security standards are also undergoing constant change, often upgraded more quickly than traditional security solutions can keep up. You need advanced security tools to help manage this constantly changing environment.
By default, Kubernetes assigns an IP address to each pod in the cluster to provide IP-based security. However, it includes only basic security measures, which means administrators must handle advanced security monitoring and compliance enforcement, usually using third-party tools designed to secure the Kubernetes stack.
Advanced Kubernetes security tools offer various capabilities, including detecting vulnerabilities and ensuring each container runs as intended. Container security processes must run continuously to protect the container host, its management stack, and its network traffic while monitoring the build pipeline’s integrity and the application’s security.
The following are some of the main risks facing Kubernetes production deployments.
To ensure the security of images, organizations should implement strong governance policies that ensure images are securely built and stored in trusted registries. For example, organizations should make sure that container images are built using pre-approved and secure base images, which should be regularly scanned for issues and vulnerabilities.
Organizations should standardize registries by creating a list of image registries that are allowed for use. Images should always be scanned before being used to create containers in a Kubernetes cluster, to avoid tampering.
Containers and pods need to communicate with each other as part of normal operations. However, this communication can be exploited by threat actors. A breached container can affect other containers and pods.
To ensure communication is secured, organizations should put in place network policies that limit communication to the minimum necessary for workloads to function. This includes both north-south traffic (ingress/egress traffic) and east-west traffic within the cluster. Network policies should be automatically adjusted to ensure they do not hurt productivity.
Visibility is critical to ensure security is maintained. However, it can be challenging to achieve visibility in complex, distributed, containerized environments.
Securing applications without visibility can be difficult or even impossible. Without visibility, it is not possible to detect critical vulnerabilities or discover misconfigurations before threat actors exploit them. Visibility is also critical to monitor containers at runtime to detect attacks, and to keep track of containers that are no longer in use in order to properly retire them before they become a liability.
Kubernetes is built to speed up the deployment of applications and simplify operations and management. While Kubernetes provides a wide range of controls that can help organizations effectively secure clusters and applications, it does not provide secure configurations out of the box.
For example, Kubernetes network policies behave similarly to firewall rules, controlling how pods can communicate with each other as well as other endpoints. Once a pod is assigned a network policy, it is only allowed to communicate with the assets declared in the network policy. However, Kubernetes does not apply network policies to pods by default. This means that after being deployed, all pods can talk to all other pods within a Kubernetes environment. This makes it critical to ensure all cluster resources have appropriate security policies defined.
Another concern is secret management. Secrets define how sensitive information, like keys and credentials, is accessed and stored. When managing secrets, it is critical to make sure that they are not used as environment variables, or hardcoded within images. Secrets should be managed externally to containers, with careful access control to protect secrets from unauthorized parties.
For many organizations, compliance is a critical concern. Achieving compliance in cloud-native environments is a highly challenging endeavor. To achieve compliance, organizations are usually required to implement certain security measures. This often requires enforcing best practices, benchmarks, and industry standards, as well as internal organizational policies.
In addition to maintaining compliance on a regular basis, organizations are also required to provide proof of compliance. This requires visibility, monitoring, and logs—the data needed to perform audits. To ensure auditors have access to the data or the systems, organizations need robust features that are not necessarily provided by plain, open-source Kubernetes.
Learn more in our guides to Kubernetes vulnerabilities and Kubernetes security issues
Kubernetes provides several concepts and mechanisms that can help secure your clusters. Here are the most important ones, as listed in the official Kubernetes project documentation.
Kubernetes uses a flat network model that allows each pod to communicate with any other pod in the cluster by default. This creates major security concerns, because it allows attackers who compromise one pod to freely communicate with all other resources in the cluster. To secure pod-to-pod communications, Kubernetes uses the concept of network policies.
A network policy is like a virtual firewall set up within each pod. When network policies are defined by an administrator, they are automatically updated at the pod level, and the pod accepts or rejects network traffic according to the policy. In Kubernetes, the mechanism for defining allowed or disallowed communication is label selectors, not IP addresses or ranges.
Kubernetes originally used the PodSecurityPolicy (PSP) object to control security-related pod configuration. A PSP defines the minimal conditions a pod must meet in order to run in the cluster. However, PSP has been found to be difficult to use, has important functional limitations, and cannot be enabled by default, and so will soon be deprecated and replaced.
The replacement feature has the temporary name “PSP Replacement Policy”. The new policy will be designed for simplicity of deployment, will have a built-in admission controller, and will be easy to use while providing flexibility for large-scale production deployments. Importantly, unlike the current PSP, you will be able to retrofit it to existing clusters via soft rollout.
Secrets are sensitive information like authentication tokens, SSH keys, or passwords. It is dangerous to store secrets in plaintext as part of container images or pod configurations, as these can easily be compromised by attackers who then gain access to any system accessible via those credentials.
The built-in Kubernetes Secrets feature lets you store sensitive information within a secret object in Kubernetes. You must first create a secret, populate it with secret data, update a service account to reference the secret, and only then create pods that use the secret. A pod can access a secret as an environment variable or a file (this is known as a secret volume).
Kubernetes uses the concept of ClusterRoles and Roles, which specify what each user can perform in a cluster or an entire Kubernetes namespace. You can use ClusterRoleBinding to apply a role to all resources in the cluster, or RoleBinding to apply the role to every resource in a namespace. Kubernetes offers the following default user-facing roles, and also lets you create additional custom roles:
The primary access point for a Kubernetes cluster is the Kubernetes API. Users and service roles access it via the kubectl utility, direct REST API requests, or client SDKs. By default, the API is accessed via port 445 and is protected by Transport Layer Security (TLS).
The Kubernetes API authentication process works as follows:
username, and can use this username in subsequent activities.Kubernetes Ingress is an object that allows services in a Kubernetes cluster to be accessed from outside the cluster. It facilitates connections via HTTP/S, and specifies routing rules.
To secure an Ingress object, you specify a secret containing two keys—tls.crt and tls.key—which contain the certificate and private key, respectively. TLS encryption works up to the ingress point, and traffic within the cluster is not encrypted by default. If the ingress configuration includes multiple hosts, they are multiplexed on the same port.
Kubernetes automatically assigns a Quality of Service (QoS) class to pods, which helps optimize scheduling and eviction of pods. The kubelet uses a QoS class to decide the order in which pods are evicted from nodes, and to make smarter decisions when scheduling pods on nodes. DevOps teams can customize QoS classes for pods if necessary.
You can place pods in a Guaranteed QoS class, which means that each container is guaranteed to receive enough resources from the node. This requires that all containers have a CPU and memory limit. Kubernetes assigns exactly enough CPU and memory resources to meet the requirements of containers in the pod.
Tigera’s commercial solutions provide Kubernetes security and observability for multi-cluster, multi-cloud, and hybrid-cloud deployments. Both Calico Enterprise and Calico Cloud provide the following features for security and observability:
Security
Observability
Next steps:
Together with our content partners, we have authored in-depth guides on several other topics that can also be useful as you explore the world of Kubernetes.
Authored by Komodor
Get updates on blog posts, workshops, certification programs, new releases, and more!
