TeamTNT: Latest TTPs targeting Kubernetes (Q1-2021)

In April 2020, MalwareHunterTeam found a number of suspicious files in an open directory and posted about them in a series of tweets. Trend Micro later confirmed that these files were part of the first cryptojacking malware by TeamTNT, a cybercrime group that specializes in attacking the cloud—typically using a malicious Docker image—and has proven itself to be both resourceful and creative.

Since this first attack, TeamTNT has continuously evolved its tactics and added capabilities to expand and capture more available cloud attack surfaces. They started with targeting exposed Docker instances and quickly added support for different C2 mechanisms, encryption, DDoS, evasion, persistence and more. Now, their latest variant is targeting the most popular container orchestrator, Kubernetes. Let’s take a closer look.

Evolving Tactics, Techniques and Procedures (TTPs)

TeamTNT’s initial attack targeted an exposed, unprotected Docker API on the internet in order to run an Alpine Linux container. Once the container started running on the unprotected Docker API, a series of scripts were downloaded to facilitate the installation of a Monero cryptominer (to carry out scanning and cleaning activities). A notable script used in the attack was <clean.sh>, which removed a bit of technically advanced Kinsing malware. Kinsing is a competing cybercrime group, so it’s notable that TeamTNT made an effort to find and remove a competing group’s attempt to target the same attack surface. Another interesting script was an AWS credential-stealing script.

With relative ease, TeamTNT quickly exploited the broader attack surface available in the cloud and started implementing new capabilities, with the end goal being cryptomining. The second variant of this malware by TeamTNT was Cetus, and was first seen in August 2020. This variant introduced additional evasion capabilities by posing as Portainer, a legitimate UI management tool.

Realizing that most containers can mount secrets in memory, TeamTNT launched a third variant of this malware, Black-T, which implemented memory password dumping like Mimikatz using mimipy and mimipenguin. Additionally, it used backdoor implementations via Weave Scope, a legitimate container-monitoring tool. First seen in October 2020, this third variant established a trend: TeamTNT hiding in plain sight using tools already available to the end user. The Black-T variant also introduced binary (/usr/bin/kube) that turned out to be an Internet Relay Chat (IRC) botnet (TNTbotinger) capable of DDoS.

Over time, TeamTNT extended attack functionality across these variants, adding support for AES and Blowfish encryption algorithms with hard-coded keys for ELF binaries, in-memory decryption and execution, and packers to evade endpoint detection solutions. This clearly shows the evolution of TeamTNT’s capabilities and a strong intent to develop new capabilities and grab a larger share of cryptomining operations. The following table shows each variant’s capabilities and intent.

Over the past year, protecting an exposed Docker API has become easier due to efforts by the security community industrywide. As a result, the opportunity and overall attack surface for this type of attack has been reduced and it has become difficult for cryptomining actors like TeamTNT to continue. Industry response forced TeamTNT to innovate and exploit the next logical (and most popular) container orchestration platform: Kubernetes. A new variant, called Hildegard, was discovered in February 2021.

Hildegard TTPs targeting Kubernetes

Hildegard is the most sophisticated malware currently targeting Kubernetes. It was discovered by Palo Alto in early 2021. It tries to find exposed kubelet APIs with anonymous access to establish a foothold in a Kubernetes cluster, and subsequently deploys cryptocurrency miners on as many pods as possible. Let’s take a look at the latest TTPs TeamTNT is using to target Kubernetes using Hildegard.

Exploiting known weaknesses

In the ‘Kubernetes Against Itself’ section of our previous post, Kubernetes Q3-2020: Threats, Exploits and TTPs, we talked about how the undocumented kubelet API can be used to run commands on default Kubernetes installations, thus providing the remote code execution (RCE) primitive to the attacker. TeamTNT was able to weaponize this weakness with its new Hildegard variant.

Mass scanning

TeamTNT still relies on the masscan utility to find vulnerable instances of kubelet on the internet. Mass internet scanning is part of TeamTNT’s general technique: identifying previous attack surfaces where they targeted Docker APIs. After compromising a vulnerable Kubernetes cluster, they continue to pivot and scan using that cluster.

Establishing C2 using a legitimate tool

Continuing with the trend of using generally available tools like Portainer and Weave Scope, TeamTNT is using tmate, a remote terminal sharing utility for Linux, to establish Command and Control (C2). It also uses a previously developed IRC module to establish additional C2 using ZiggyStartux.

Using third-party integration to aid process evasion

TeamTNT uses a common Linux process name, bioset, to disguise the malicious process. It also uses a library injection technique using libprocesshider, which uses LD_PRELOAD to load a malicious library when any process runs on the system. The library overwrites the readdir() and readdir64() functions to filter malicious processes from showing up in programs like ‘ps’ or ‘lsof’. This capability is a major upgrade from previous evasion approaches, as cloud operators will now need kernel-based eBPF monitoring to detect such processes.

Bypassing coreDNS

If you have DNS monitoring in place with coreDNS in the cluster, it can easily be bypassed by TeamTNT because they overwrite the DNS servers from the /etc/resolve.conf file and re-route traffic to a preferred DNS server (usually 8.8.4.4 and 8.8.8.8). The best defense against this technique is to use default deny policies and whitelist DNS traffic.

Using red teaming tools

TeamTNT uses red teaming tools from projects like Peirates and BOtB that have container breakout and information scraping capabilities. Use of these tools, in addition to libprocesshider and tmate, suggests that TeamTNT wants to leverage existing functionality rather than develop their own. It also shows the skill and sophistication of this actor, highlighting their ability to rely on third-party tools to quickly add capabilities they need. We expect this to change in future as this group matures and adds its own custom-built code.

Mitigation

Disable kubelet’s unauthorized access

The easiest and most powerful mitigation for this attack is to secure your kubelet instances by setting –anonymous-auth argument to false. You can use –authorization-mode flag to enable role-based access control (RBAC) and –client-ca-file flag to be stringent. Additionally, you can disable port 10255 by setting –read-only-port to ‘0’.

Deny ingress traffic to kubelet on nodes

You can use the below hostEndpoint based policy from Calico Enterprise to secure your nodes by blocking external access to the kubelet. With this policy, only localhost can access its own kubelet service and the master node can access the worker node’s kubelet service.

The policy is a two-step process. Step 1 involves enabling the automatic hostEndpoint using the commands outlined below, which create hostEndpoints for all the nodes in the cluster (label them as ‘kubernetes-host’). If you are using Calico, you can also create them manually using this guide. In step 2, we block all access to kubelet except from the master and worker nodes.

Step 1: Enable automatic hostEndpoint creation and label each node.

# Enable hostendpointcreation
# kubectl patch kubecontrollersconfiguration default --patch='{"spec": {"controllers": {"node": {"hostEndpoint": {"autoCreate": "Enabled"}}}}}'

# label all nodes
# kubectl label nodes --all kubernetes-host=

Step 2: Secure kubelet service on nodes using policy. Note that your dependent functionality with kubelet needs to be whitelisted in this policy.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: ingress-k8s-workers
spec:
selector: has(kubernetes-host)
# Allow all traffic to localhost.
ingress:
- action: Allow
destination:
nets:
- 127.0.0.1/32
# Allow only the masters access to the nodes kubelet API.
- action: Allow
protocol: TCP
source:
selector: has(node-role.kubernetes.io/master)
destination:
ports:
- 10250

Block all DNS traffic except coreDNS

Make sure you have a policy to allow only DNS traffic to coreDNS service in the cluster. It will block any unintended DNS traffic from leaving your cluster. Calico Enterprise uses a tiered policy to allow only coreDNS traffic, and blocks all other DNS traffic with a ‘Pass’ action, as is shown below.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: dns-deny.allow-coredns
spec:
tier: dns-deny
selector: all()
egress:
- action: Pass
protocol: UDP
destination:
selector: k8s-app == "kube-dns"
ports:
- '53'
types:
- Egress

Summary

From its first attack to its latest variant targeting Kubernetes, TeamTNT has nimbly evolved to become a prominent threat actor targeting the cloud. Currently, the goal of their attacks appears to be cryptojacking compromised containers. As this group matures and adds new capabilities, they are expected to carry out data exfiltration activities from a compromised organization, as is popular among many APT groups. We definitely expect to see more sophistication and new techniques from this cybercrime group in future.

To learn more about the latest TTPs being used by threat actors to target Kubernetes clusters, register for our upcoming webinar.