Kubernetes has become the world’s most popular container orchestration system and is taking the enterprise ecosystem by storm. At this disruptive moment it’s useful to look back and review the security threats that have evolved in this dynamic landscape. Identifying these threats and exploits and being a proactive learner may save you a lot of time and effort…as well as help you retain your reputation in the long run. In this blog we’ll look at some critical security issues faced by the Kubernetes ecosystem in the recent past, and examine the top tactics, techniques and procedures (TTPs) used by attackers.
Everyday, new Kubernetes ecosystem Common Vulnerabilities and Exposures (CVEs) are published. Let’s take a closer look at some of the cloud shakers…
CVE-2020-14386: Using privilege escalation vulnerability to escape the pod
A flaw was found in the Linux kernel before 5.9-rc4. Memory corruption can be exploited to gain root privileges from unprivileged processes.
We received notification that some instances in our cloud infrastructure are vulnerable to this CVE. When we took a closer look, it appeared to be a typical privilege escalation vulnerability using AF sockets on hosts. Unprivileged users with CAP_NET_RAW permissions can send packets to the local socket and try to exploit the vulnerability in Linux kernels before 5.9-rc4 and gain root privileges. In cloud context, this mechanism becomes alarming as pods use node sockets for their operation and by default Kubernetes provides CAP_NET_RAW permission to the pods. Now with this context, if an attacker exploits this flaw successfully from or within the pod, the attacker will be able to gain privileges on the node and escape the Kubernetes pod. This has significant adverse and unintended consequences. The root of the issue, CAP_NET_RAW permission, has been a source of many Kubernetes networking security issues in the past. You avoid this permission using PodSecurityPolicy on the cluster.
CVE-2020-2121: Kubernetes CI/CD plugin for remote code execution
Jenkins Google Kubernetes Engine Plugin 0.8.0 and earlier does not configure its YAML parser to prevent the instantiation of arbitrary types, resulting in a remote code execution vulnerability.
Kubernetes CI/CD plugins are turning out to be a great target for attackers. A relatively new plugin ecosystem is often exposed by flaws like CloudShare and Arquillian vulnerabilities. CVE-2020-2121 is the latest remote code execution in this plugin, which publishes deployments built with it to the cluster. The flaw is in the YAML parser which allows instantiation of arbitrary types to allow remote code execution on the pod. The plugin stores its password file on the pod itself which is easy to retrieve using this vulnerability, and the attacker gets control of the Jenkins.
The role used by the plugin has ‘CREATE’ pod privileges on the cluster, shown here. The attacker can schedule pods on the cluster and create services or endpoints. If PSP (PodSecurityPolicy) is not applied, attackers can create pods with hostnetwork to attack the underlay network, as well as steal credentials stored on nodes using volume mounts and begin their lateral movement journey.
CVE-2020-8558: Flaw in kubelet and kube-proxy and CAP_NET_RAW permission
The Kubelet and kube-proxy components in versions 1.1.0-1.16.10, 1.17.0-1.17.6, and 1.18.0-1.18.3 were found to contain a security issue which allows adjacent hosts to reach TCP and UDP services bound to 127.0.0.1 running on the node or in the node’s network namespace.
This is one of the vulnerabilities that received some media limelight. This flaw in kubelet and kube-proxy allows services bound to 127.0.0.1 localhost ip to be accessed by adjacent hosts. For me, it is another example which proves analyzing a vulnerability in cloud context is crucial. When the CVE was published back in April the impact assessment was medium. Unit 42 discovered that a few Kubernetes installations don’t disable the API server’s insecure-port by default, which is only accessible within the master node. With this vulnerability, attackers were able to reach it via CAP_NET_RAW privileges from the host or from a pod running on the same host to compromise the larger cluster. If you have been running a Kubernetes cluster for a long time, it might be a good idea to check if you have an insecure port open as described here.
CVE-2020-8559: Abusing cluster API server requests
The Kubernetes kube-apiserver in versions v1.6-v1.15, and versions prior to v1.16.13, v1.17.9 and v1.18.6 are vulnerable to an unvalidated re-direct on proxied upgrade requests that could allow an attacker to escalate privileges from a node compromise to a full cluster compromise.
This is one of the more interesting CVEs, whereby if a Kubernetes is compromised, it can redirect cluster API server requests (to port-forward, attach, exec) using HTTP response code. Once an API server receives a redirect response code from a compromised node, it sends the request to a new URL, which could be to (a) a pod on a different node or (b) an attacker-controlled Kubernetes cluster if CN authority is the same to verify SSL. This request contains privileged credentials along with the original request. The mechanics of the exploit are enough to illustrate how Kubernetes components can be used to gain a larger compromise.
Honorary mentions go to CVE-2020-10749, which is an attack using rogue IPv6 advertisement to disrupt routing planes. And CVE-2020-8557, in which the pod can DoS the node using a kublet-mounted /etc/hosts file which doesn’t have a size restriction.
Major Attacks and TTPs
Cloud practitioners are armed with best-in-class knowledge, support and security practices. Still, statistically it is impossible to have a similar security posture for all cloud instances world-wide. And attackers do take advantage of this statistical improbability. They are using evolved tactics, techniques and procedures (TTPs) to exploit edge cases, which have led to many breaches like Capital One, Jenkins, and Docker. Here we will look at some of the top TTPs that attackers have recently used.
Finding these Docker instances is as simple as probing open ports (2375, 2376, 2377, 4243, 4244) on the Internet or search engines like Shodan. Notorious groups like TeamTNT are seen targeting these instances and have also started using legit Kubernetes monitoring tools like Weave Scope to backdoor these Docker instances. Numerous crypto-jacking malwares like Cetus, Graboid, and campaigns exploiting CI/CD tools are taking advantage of Docker configuration flaws. DDoS malwares like XORDDoS and Kaiji also targeted vulnerable Docker instances and used those resources to increase their DDoS capacity. If you have Docker in your environment, making use of cloud provider firewalls (security groups in AWS) is essential. Having basic monitoring capabilities to check running containers, CPU utilization, and network activity is worth it in such cases. Additionally, embedding threat intelligence into the cloud and monitoring various thresholds for security violations can help security teams tremendously.
Malicious Docker Images
Public image repositories like Docker hub are turning into an easy medium to distribute malicious images under the name of custom configurations and usability. The Docker incident where a malicious image was pulled 5 million times before being removed indicates the reach of this attack vector. It’s paramount to check sources of your base images if you are creating your own or using images straight from a public repository. The image history can be checked using Docker or forensic analysis tools like dive. The screenshot, below, shows the infamous DzMLT Docker hub account which distributed the XMRig miner, primarily mining Monero. It clearly shows XMRig miner being downloaded and configured into image.
We trust cloud providers to configure, patch and manage Kubernetes security on our behalf. But the devil is always in the details. As Christophe Tafani-Dereeper showed in this blog post, even in these platforms certain configurations can lead to privilege escalation and facilitate the take-over of the cluster. The author’s approach was to retrieve an IAM role credential by reaching the metadata service from a compromised pod. This IAM role had too many permissions by default, which allowed privilege escalation in a managed offering. Cloud security teams should build expertise and audit their managed environments so that such holes can be detected and plugged permanently.
Kubernetes Against Itself
Kubernetes components like kubelet, controller-manager, etcd, and others can be exploited to obtain a higher level of privileges and persistence in the cluster. Unauthenticated kubelet’s undocumented API can be used for remote code execution inside a container as shown in this example. Rhino Security Labs showed how an attacker can join as a fake worker node There are numerous ways that Kubernetes can be used against itself, here are a few common examples:
- Run a second kubelet on a node so that the same node will be part of two different clusters. One will be attacker controlled.
- Privileged pods can provide root access on a node
- Hostnetwork pod allows it to attack underlay network
- Insecure root mounts
- Webhook can be used to intercept any API request
- Network stack can be exploited to attack VXLAN and IPIP networks
Cloud infrastructure is a black box, but still needs oversight to ensure it has security and the right tools for DevSecOps teams to do their jobs. Having a system to monitor your Kubernetes clusters and a mechanism to baseline the behavior can go a long way in facilitating detection, investigation and mitigation of threats. As we showed in this blog, the cloud adds a new dimension to an organization’s attack surface. A classic CVE can become even more relevant in the cloud. Cloud security teams need to understand the impact of each of these CVEs and apply mitigation as soon as possible. At the same time, attackers are using evolved TTPs to exploit, control and maintain persistent access in compromised cloud infrastructure. To thwart attackers, organizations must bolster their monitoring and cloud visibility capabilities, regularly audit their environment, and apply the latest security patches as they become available. Using the most recent advances in data science and machine learning techniques to detect attacker behavior is essential. Calico Enterprise’s threat defense capabilities are targeting attackers with advanced tools that can help security teams develop best-in-class detection capabilities in the cloud.
Free Online Training
Access Live and On-Demand Kubernetes Training
Calico Enterprise – Free Trial
Network Security, Monitoring, and Troubleshooting
for Microservices Running on Kubernetes