Kubernetes Security: Risks, Security Controls, and Best Practices

 

Is Kubernetes Secure?

Kubernetes is growing in popularity, and its use within organizations is maturing. According to the latest Cloud Native Computing Foundation (CNCF) annual survey, 83% of organizations surveyed use Kubernetes in production, and 28% of organizations have more than 11 Kubernetes production clusters.

At the same time, security concerns around Kubernetes are mounting. A Red Hat survey of Kubernetes adoption and security showed that, of 500 DevOps professionals surveyed:

• 55% delayed an application release due to security issues.
• 94% experienced at least one Kubernetes security incident in the past year.
• 59% said security is their biggest concern with regard to continued use of Kubernetes and containers.

Read on to better understand the security risks facing Kubernetes environments, understand what security mechanisms Kubernetes natively provides, and discover best practices for improving your Kubernetes security posture.

In this article, you will learn:

• Kubernetes Security Risks
  • Compromised Images and Image Registries
  • Compromised Containers or Malicious Traffic
  • Lack of Visibility
  • Unsecure Default Configurations
  • Compliance Challenges
• The 4 C’s of Cloud-Native Security
• Security Mechanisms in a Kubernetes Cluster
  • Network Policy
  • Pod Security Policy
  • Kubernetes Secrets
  • Kubernetes RBAC
  • Authentication
  • TLS For Kubernetes Ingress
  • Quality of Service
• Kubernetes Security Best Practices
  • Build-Time Security
  • Deploy-Time Security
  • Runtime Security
• Kubernetes Security and Observability with Tigera

Kubernetes Security Risks

The following are some of the main risks facing Kubernetes production deployments.

Compromised Images and Image Registries

To ensure the security of images, organizations should implement strong governance policies that ensure images are securely built and stored in trusted registries. For example, organizations should make sure that container images are built using pre-approved and secure base images, which should be regularly scanned for issues and vulnerabilities.

Organizations should standardize registries by creating a list of image registries that are allowed for use. Images should always be scanned before being used to create containers in a Kubernetes cluster, to avoid tampering.

Compromised Containers or Malicious Traffic

Containers and pods need to communicate with each other as part of normal operations. However, this communication can be exploited by threat actors. A breached container can affect other containers and pods.

To ensure communication is secured, organizations should put in place network policies that limit communication to the minimum necessary for workloads to function. This includes both north-south traffic (ingress/egress traffic) and east-west traffic within the cluster. Network policies should be automatically adjusted to ensure they do not hurt productivity.

Lack of Visibility

Visibility is critical to ensure security is maintained. However, it can be challenging to achieve visibility in complex, distributed, containerized environments.

• There can be a large number of containers scheduled, deployed, and terminated—all of which need to be tracked, monitored, and managed.
• The distributed and dynamic nature of a containerized workload makes it difficult to collect, track, and understand relevant metrics.
• Kubernetes can be deployed in multi-cloud or hybrid cloud environments. Each cloud vendor provides its own monitoring and visibility tools, and it becomes very difficult to allow consistent visibility across environments.

Securing applications without visibility can be difficult or even impossible. Without visibility, it is not possible to detect critical vulnerabilities or discover misconfigurations before threat actors exploit them. Visibility is also critical to monitor containers at runtime to detect attacks, and to keep track of containers that are no longer in use in order to properly retire them before they become a liability.

Unsecure Default Configurations

Kubernetes is built to speed up the deployment of applications and simplify operations and management. While Kubernetes provides a wide range of controls that can help organizations effectively secure clusters and applications, it does not provide secure configurations out of the box.

For example, Kubernetes network policies behave similarly to firewall rules, controlling how pods can communicate with each other as well as other endpoints. Once a pod is assigned a network policy, it is only allowed to communicate with the assets declared in the network policy. However, Kubernetes does not apply network policies to pods by default. This means that after being deployed, all pods can talk to all other pods within a Kubernetes environment. This makes it critical to ensure all cluster resources have appropriate security policies defined.

Another concern is secret management. Secrets define how sensitive information, like keys and credentials, is accessed and stored. When managing secrets, it is critical to make sure that they are not used as environment variables, or hardcoded within images. Secrets should be managed externally to containers, with careful access control to protect secrets from unauthorized parties.

Compliance Challenges

For many organizations, compliance is a critical concern. Achieving compliance in cloud-native environments is a highly challenging endeavor. To achieve compliance, organizations are usually required to implement certain security measures. This often requires enforcing best practices, benchmarks, and industry standards, as well as internal organizational policies.

In addition to maintaining compliance on a regular basis, organizations are also required to provide proof of compliance. This requires visibility, monitoring, and logs—the data needed to perform audits. To ensure auditors have access to the data or the systems, organizations need robust features that are not necessarily provided by plain, open-source Kubernetes.

 

Learn more in our guide to Kubernetes vulnerabilities

 

The 4 C’s of Cloud-Native Security

Cloud-native security is implemented across four key layers, known as the 4 C’s:

• Code – Security measures that protect the code. For example, using vulnerability scanners and secure coding practices.
• Containers – Security measures at the container level. For example, restricting access to network ports and encrypting data-in-transit.
• Clusters – Security measures at the cluster level. For example, defining network security policies and hardening all master nodes.
• Cloud or enterprise data centers – Security measures that protect infrastructure. This is usually implemented by cloud providers or on-premises IT staff.

 

Security Mechanisms in a Kubernetes Cluster

Kubernetes provides several concepts and mechanisms that can help secure your clusters. Here are the most important ones, as listed in the official Kubernetes project documentation.

Network Policy

Kubernetes uses a flat network model that allows each pod to communicate with any other pod in the cluster by default. This creates major security concerns, because it allows attackers who compromise one pod to freely communicate with all other resources in the cluster. To secure pod-to-pod communications, Kubernetes uses the concept of network policies.

A network policy is like a virtual firewall set up within each pod. When network policies are defined by an administrator, they are automatically updated at the pod level, and the pod accepts or rejects network traffic according to the policy. In Kubernetes, the mechanism for defining allowed or disallowed communication is label selectors, not IP addresses or ranges.

Pod Security Policy

Kubernetes originally used the PodSecurityPolicy (PSP) object to control security-related pod configuration. A PSP defines the minimal conditions a pod must meet in order to run in the cluster. However, PSP has been found to be difficult to use, has important functional limitations, and cannot be enabled by default, and so will soon be deprecated and replaced.

The replacement feature has the temporary name “PSP Replacement Policy”. The new policy will be designed for simplicity of deployment, will have a built-in admission controller, and will be easy to use while providing flexibility for large-scale production deployments. Importantly, unlike the current PSP, you will be able to retrofit it to existing clusters via soft rollout.

Kubernetes Secrets

Secrets are sensitive information like authentication tokens, SSH keys, or passwords. It is dangerous to store secrets in plaintext as part of container images or pod configurations, as these can easily be compromised by attackers who then gain access to any system accessible via those credentials.

The built-in Kubernetes Secrets feature lets you store sensitive information within a secret object in Kubernetes. You must first create a secret, populate it with secret data, update a service account to reference the secret, and only then create pods that use the secret. A pod can access a secret as an environment variable or a file (this is known as a secret volume).

Kubernetes RBAC

Kubernetes uses the concept of ClusterRoles and Roles, which specify what each user can perform in a cluster or an entire Kubernetes namespace. You can use ClusterRoleBinding to apply a role to all resources in the cluster, or RoleBinding to apply the role to every resource in a namespace. Kubernetes offers the following default user-facing roles, and also lets you create additional custom roles:

• Cluster-admin – Can perform any action on anything in a cluster.
• Admin – Allows unlimited read/write access to resources in a namespace, but does not provide write access to the namespace itself.
• Edit – Allows read/write access within a namespace, but does not allow viewing or modifying roles and bindings.
• View – Like Edit, but allows read-only access in a namespace.

Authentication

The primary access point for a Kubernetes cluster is the Kubernetes API. Users and service roles access it via the kubectl utility, direct REST API requests, or client SDKs. By default, the API is accessed via port 445 and is protected by Transport Layer Security (TLS).

The Kubernetes API authentication process works as follows:

1. The client attempts to access the API, presents a TLS client certificate, and a TLS connection is established.
2. The API server runs one or more Authenticator modules. These allow cluster administrators to define user and service accounts and specify authentication strategies, including client certificates, tokens, JSON Web Tokens, and passwords.
3. The Authentication modules accept the client’s entire HTTP request and attempt to authenticate it. If there are several authentication modules, they are tried one by one until one succeeds.
4. If authentication fails, the API returns status code 401. If it succeeds, the client is authenticated as a username, and can use this username in subsequent activities.
5. The Authentication module might assign the user group membership. If so, the user will have the additional privileges of the group.

TLS For Kubernetes Ingress

Kubernetes Ingress is an object that allows services in a Kubernetes cluster to be accessed from outside the cluster. It facilitates connections via HTTP/S, and specifies routing rules.

To secure an Ingress object, you specify a secret containing two keys—tls.crt and tls.key—which contain the certificate and private key, respectively. TLS encryption works up to the ingress point, and traffic within the cluster is not encrypted by default. If the ingress configuration includes multiple hosts, they are multiplexed on the same port.

Quality of Service

Kubernetes automatically assigns a Quality of Service (QoS) class to pods, which helps optimize scheduling and eviction of pods. The kubelet uses a QoS class to decide the order in which pods are evicted from nodes, and to make smarter decisions when scheduling pods on nodes. DevOps teams can customize QoS classes for pods if necessary.

You can place pods in a Guaranteed QoS class, which means that each container is guaranteed to receive enough resources from the node. This requires that all containers have a CPU and memory limit. Kubernetes assigns exactly enough CPU and memory resources to meet the requirements of containers in the pod.

Kubernetes Security Best Practices

Here are a few best practices that can help you secure Kubernetes clusters and workloads throughout the software development lifecycle.

Build-Time Security

• Image scanning – Ensure container images are free of vulnerabilities, by applying automated image scanning at every stage of the CI/CD pipeline. Check images both for CVE vulnerabilities and insecure configuration, such as hard coded secrets.
• Host operating system hardening – Ensure containers have minimal privileges on the host, and harden the operating system to restrict system calls, and ensure strong isolation between containers.
• Base container images – To minimize the attack surface, when creating a container image, you should create it from scratch. If you do use base images, select a minimal image that only has the libraries or operating system components your containers absolutely need.

 

Learn more in our blog post: What Your Kubernetes Security Checklist Might Be Missing

 

Deploy-Time Security

• Kubernetes cluster hardening – Use automated tools to review your cluster configuration for security best practices, and remediate any insecure configuration. Set up RBAC and TLS for data in transit.
• Network security integration with Kubernetes clusters – You can feed IP addresses used by Kubernetes workloads to external tools like next generation firewalls (NGFW), to make them aware of the Kubernetes environment. In addition, define security groups on cloud providers to limit communication within and between clusters.

Runtime Security

• Network security controls within the cluster – Build network security definitions into your workloads, using a declarative approach. Anywhere a workload runs, it must take its security definitions with it. You can do this using a Kubernetes-native network and security policy solution like Calico, which restricts traffic at the network level (Layer 3-4) and integrates with Envoy (a proxy) to restrict traffic at the application level (Layer 7).
• Establishing enterprise security controls – Secure Kubernetes clusters like you would any enterprise workload. Enable encryption of data in transit, enable auditing and compliance reports, and perform continuous compliance checks with automated remediation.
• Threat defense – Establish intrusion detection and prevention measures, and gain visibility over suspicious activity and threats, aggregating log data by groups of similar pods. Use machine learning analysis and threat intelligence to derive insights from logs and detect threats faster and more effectively.

 

Related content: Kubernetes security policy design best practices

 

Kubernetes Security and Observability with Tigera

Tigera’s commercial solutions provide Kubernetes security and observability for multi-cluster, multi-cloud, and hybrid-cloud deployments. Both Calico Enterprise and Calico Cloud provide the following features for security and observability:

Security

• Security policy preview, staging, and recommendation – Easily make self-service security policy changes to a cluster without the risk of overriding an existing policy. Calico can auto-generate a recommended policy based on ingress and egress traffic between existing services, and can deploy your policies in a “staged” mode before the policy rule is enforced.
• Compliance reporting and alerts – Continuously monitor and enforce compliance controls, easily create custom reports for audit.
• Intrusion detection & prevention (IDS/IPS) – Detect and mitigate Advanced Persistent Threats (APTs) using machine learning and a rule-based engine that enables active monitoring.
• Microsegmentation across Host/VMs/Containers – Deploy a scalable, unified microsegmentation model for hosts, VMs, containers, pods, and services that works across all your environments.
• Data-in-transit encryption – Protect sensitive data and meet compliance requirements with high-performance encryption for data-in-transit.

Observability

• Dynamic Service Graph – Get a detailed runtime visualization of your Kubernetes environment to easily understand microservice behavior and interaction.
• Application Layer Observability – Gain visibility into service-to-service communication within your Kubernetes environment, without the operational complexity and performance overhead of service mesh.
• Dynamic Packet Capture – Generate pcap files on nodes associated with pods targeted for packet capture, to debug microservices and application interaction.
• DNS Dashboard – Quickly confirm or eliminate DNS as the root cause for microservice and application connectivity issues in Kubernetes.
• Flow visualizer – Get a 360-degree view of a namespace or workload, including analytics around how security policies are being evaluated in real time and a volumetric representation of flows.

Next steps:

• Read our guide: Kubernetes observability
• Read our O’Reilly book: Kubernetes Security and Observability (free download)