Five Ways to Quickly Uncover Malicious Activity and Protect Your Kubernetes Workloads

Organizations are rapidly moving more and more mission-critical applications to Kubernetes (K8s) and the cloud to reduce costs, achieve faster deployment times, and improve operational efficiencies, but are struggling to achieve a strong security posture because of their inability to apply conventional security practices in the cloud environment. Commitment to cloud security grows, but security safeguards are not keeping up with the increased use of the various cloud platforms. Regardless of the cloud provider or service model, individual organizations are ultimately responsible for the security of their data.

According to a 2019 Ponemon Institute Global Cloud Data Security Study, 70 percent of respondents find it more complex to manage privacy and data protection regulations in a cloud environment than on-premises. Meanwhile, the percent of corporate data stored in the cloud environment has grown from an average of 30 percent in 2015 to an average of 48 percent in 2019. In the same study, 56 percent of respondents say the use of cloud resources increases compliance risk.

The downside associated with a security breach is severe for any organization, but especially so for companies in regulated environments like financial services, healthcare and telecommunications. Now there’s a new and highly effective way for security operations teams responsible for protecting Kubernetes workloads to gain greater control of their cloud security posture while reducing operational complexity.

Using Calico Enterprise with Global Alerts, a new feature just released, organizations can easily monitor critical Kubernetes workloads in any cloud environment, quickly identify malicious behavior, and significantly reduce dwell time.

Calico Enterprise can be used to detect:

  1. Lateral movement by the attacker for East-West and North-South traffic
    • You can monitor all flows within your cluster for threat and lateral movement detection
  2. Botnet, Command and Control, VPN-TOR
    • You can integrate intelligence feeds with Calico Enterprise using Global Alerts threat feeds
  3. Malicious DNS queries
    • Using suspicious domain feeds, Calico Enterprise with Global Alerts flags any DNS requests to a malicious domain from your workload
  4. Cloud Metadata API attacks.
    • You can identify workload flows that attempt to connect to cloud resources like metadata API if you are on Kubernetes etcd, RBAC, etc. where an attacker attempts to enumerate access and elevate his/her privileges
  5. DGA malware traffic.
    • Using a machine learning algorithm, you can detect any malicious DGA domain traffic from your cluster. We’ll discuss this in more detail in a future blog

Calico Enterprise with Global Alerts provides advanced threat and lateral movement detection capabilities for Kubernetes workloads in the cloud, enabling you to more easily uncover malicious activity in advanced stages.

In this blog we will show you how an attacker compromises a public-facing application and gains a foothold in a cloud network. Then he goes on to find his next target within the cluster, moving laterally to further his attack campaign. At the same time we will show you how Global Alerts, a new feature within Calico Enterprise, detects these malicious activities at each stage.

Attack scenario:

We will launch a cloud infrastructure attack by compromising multiple pods and move laterally into the cloud infrastructure, taking advantage of any weaknesses. Our front-end website is using Drupal application running on Drupal-pod in default namespace, on port 80. And Webmin service is running under namespace ‘crown-space’ which is not exposed to the internet. Webmin controls and monitors server resources and is accessible on port 10000.

We will configure the Global Alerts around these critical apps so that monitoring of these resources is automated.

First, let’s review aspects of our scenario which may uncover an attacker’s behavior.

  1. Lateral movement attempt from Drupal-pod
    • Any connection originating from Drupal-pod to internal network apart from approved flows
    • Any connection made to Drupal-pod apart from its service port 80
  2. Lateral movement attempt to Webmin-pod
    • Any connection made to Webmin-pod apart from its service port 10000
    • Any connection originating from webmin-pod apart from approved flow
  3. Cloud metadata API requests from crown-space and default namespace
    • To detect user-made cloud metadata API requests which are unusual in a Kubernetes pod
  4. Threat feeds
    • Monitor any connection made to suspicious IP or domains from the Kubernetes cluster
    • Attacker or one of his pivoting tools may end up talking to suspicious IP or domain

In the next step, we configure Global alerts to detect and respond to the above conditions.

Configure GlobalAlerts :

Let’s configure GlobalAlert using following manifests.

Manifest 1:

  • Checks if there is any connection originating from Drupal-pod
  • We want to whitelist any approved flow
    • For Drupal it is port 80
    • Approved flows can be added with ‘NOT’ operator in query
apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.lateral.from-drupal
spec:
  description: "Lateral movement detected: Outgoing Connection from Drupal"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: "source_name_aggr='drupal-pod' AND proto='tcp' AND action='allow' AND reporter=src AND NOT (source_port=80) AND NOT (dest_name_aggr='kse.kubernetes') AND NOT (dest_name_aggr='pub') AND NOT (dest_port=10000) AND NOT (dest_name_aggr='metadata-api')"

Manifest 2:

  • Check if the Webmin-pod is accessed from any other port apart from 10000
  • We want to whitelist any approved flow
    • For Drupal it is port 80
    • Approved flows can be added with ‘NOT’ operator in query
  • This condition indicates an unapproved open port which could be bind shell on webmin pod
apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.lateral.to-webmin
spec:
  description: "Lateral movement detected: Webmin accessed"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: "dest_name_aggr='webmin-pod' AND proto='tcp' AND action='allow' AND reporter='src' AND NOT (dest_port=10000)"

Manifest 3:

  • Any connection originating from webmin-pod apart from port 10000
  • This condition indicates a reverse shell or unapproved activity
  • We can use ‘NOT’ operator to exclude approved flows
apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.lateral.from-webmin
spec:
  description: "Lateral movement detected: Outgoing connection from Webmin"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: "source_name_aggr='webmin-pod' AND proto='tcp' AND action='allow' AND reporter=src AND NOT (source_port=10000) AND NOT (dest_name_aggr='kse.kubernetes') AND NOT (dest_name_aggr='pub') AND NOT (dest_name_aggr='metadata-api')"

Manifest 4:

  • Any requests made to cloud provider’s metadata API from application namespace
  • This is very suspicious within a cloud environment if your namespace isn’t supposed to use this API for any legitimate activity
apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.metadata.requests
spec:
  description: "Metadata API: Connection to metadata service from user namespace"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: (dest_name_aggr='metadata-api' OR dest_name_aggr='kse.kubernetes') AND action='allow' AND proto='tcp' AND reporter=src AND (source_namespace='default' OR source_namespace='crown-space')

Manifest 5:

Configure Global Network Set for metadata API.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: metadata-api
spec:
  nets:
  - 169.254.169.254

Manifest 5a:

Suspicious Domains threat feed.

#manifest 6
apiVersion: projectcalico.org/v3
kind: GlobalThreatFeed
metadata:
  name: global.threat.domains
spec:
  content: DomainNameSet
  pull:
    http:
      url: http://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt

Manifest 5b:

Suspicious IP Threat feed. You can find tor exit node feed here and ejr-vpn feed here. These feeds can be used under pull method.

#manifest 7
apiVersion: projectcalico.org/v3
kind: GlobalThreatFeed
metadata:
  name: global.threat.ipfeodo
spec:
  pull:
    http:
      url: http://feodotracker.abuse.ch/downloads/ipblocklist.txt
  globalNetworkSet:
    labels:
      feed: feodo

Manifest 5c:

Suspicious IP feed, block policy.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: default.block-feodo
spec:
  tier: default
  selector: app == 'crown-webmin'
  types:
  - Egress
  egress:
  - action: Deny
    destination:
      selector: feed == 'feodo'
  - action: Allow

We recommend that you review the  Global Alerts documentation to understand the various fields, some of which we will discuss. All manifests run every 10 minutes and look at data T-10 minutes. (period and lookback)

 

Attack on Drupal Site:

Let’s look at a front end application on the cloud. The public drupal site is accessible on port 80 and serves the following page.

To find the version of Drupal, we will connect to the site to look at HTTP headers. The banner grab, below, indicates the installation is Drupal 8.

Drupal 8 has a ‘Form property injection’ vulnerability, i.e. CVE-2018-7600, and has an exploit module with metasploit ‘unix/webapp/drupalgeddon2’.

The exploit targets the Drupal form fields API. Unvalidated form fields on the page are used to inject malicious array into form API so that drupal AJAX callback renders array by executing it.

By running the metasploit exploit on the Drupal server, you can see a reverse shell is spawned exploiting the vulnerability successfully on port 4444. We see a Drupal daemon process is running as non-root. Though we can now copy backdooring tools like netcat and kubectl binaries on Drupal-pod.

 ./kubectl auth can-i --list
Resources                                       Non-Resource URLs        Resource Names   Verbs
globalnetworkpolicies.projectcalico.org         []                       []               [*]
networkpolicies.projectcalico.org               []                       []               [*]
stagedglobalnetworkpolicies.projectcalico.org   []                       []               [*]
stagednetworkpolicies.projectcalico.org         []                       []               [*]
selfsubjectaccessreviews.authorization.k8s.io   []                       []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                       []               [create]
                                                [/api/*]                 []               [get]
                                                [/api]                   []               [get]
                                                [/apis/*]                []               [get]
                                                [/apis]                  []               [get]
                                                [/healthz]               []               [get]
                                                [/openapi/*]             []               [get]
                                                [/openapi]               []               [get]
                                                [/swagger-2.0.0.pb-v1]   []               [get]
                                                [/swagger.json]          []               [get]
                                                [/swaggerapi/*]          []               [get]
                                                [/swaggerapi]            []               [get]
                                                [/version/]              []               [get]
                                                [/version]               []               [get]

 

We have three ways to proceed:

  1. Try to get root privileges at drupal-pod (which is not always possible) or
  2. Abuse service token in our case it doesn’t have special priviledges
  3. Find another target within the cluster and move laterally to gain greater privileges. We will see the later part of the attack in the next section.

Detection:

  • Check Global Alerts on Calico Enterprise. As soon as your felix interval is finished, you will see an alert with details of the connection, above
  • Use these details to track the flow in Elasticsearch and traffic at that time
  • We can see the global.lateral.from-drupal alert being triggered
  • At this stage an incident response team can be notified to investigate and uncover the compromise

Look at the following picture. This is the alert that was generated, when Drupal was compromised and the reverse shell was spawned.

 

Attack on Webmin server:

Once we have control of the Drupal pod, with the lack of privilege and access, we look into other exposed resources that are reachable from this pod. By enumerating the network, we see there is a webmin application running on port 10000.

Now, the HTTP banner grab confirms MiniServ 1.910 is running. The attacker now has an interesting target as this version of MiniServ is backdoored which is identified by CVE-2019-15107.

The backdoor was hidden for over a year and eventually discovered. The exploit targets the password change function where ‘old password’ is executed as command. Metasploit has a module unix/webapp/webmin_backdoor which exploits this vulnerability.

Now, to reach the Webmin application from the attacker machine, the compromised Drupal-pod can be used as a network relay.

The following command will create a relay between the attacker and Drupal-pod so that the attacker’s requests will be forwarded to the Webmin application.

We can reach the Webmin application by making requests to port 5000 locally from the attacker machine as follows.

attacker# curl -kv http://127.0.0.1:5000

 

Now we can send an exploit to Webmin through Drupal-pod and we will listen for the reverse shell on Drupal-pod on port 2222. Why? There is no way for the Webmin-application to talk to the attacker directly as it’s an internal app (which we consider as East-West traffic).

Let’s set our metasploit variables RHOST as 127.0.0.0 and RPORT as 5000 (relay to Webmin-application) and LHOST as Drupal-pod IP on attacker metasploit.

And on Drupal-pod, we want to make sure we are listening on port 2222 with the copied netcat binary.

Note that the handler fails to bind on the attacker machine as our LHOST is set as rupal-pod’s IP. But we successfully get the shell on Drupal-pod where we are listening on port 2222. That means the exploit was successfully sent to the Webmin-application.

Now we have root access on the Webmin-pod (shown in the image above) with uid=0. With root access, the attacker can use this pod to make changes to the file system and cover his tracks by deleting logs. The attacker can also install malware, miners, and stealers and also use it as a pivot to launch further attacks.

Detection:

  • Check the Global Alerts tab on Calico Enterprise to view  this lateral move from Drupal-pod to Webmin-pod
  • We will see Manifests 2 and 3 are triggered
  • We can further investigate connections, traffic, and movement at the same time using inputs from the alert

Reverse shell from Webmin to Drupal :

 

Drupal to Webmin attack connection:

Additional Alerts

ThreatFeeds:

In this example, I didn’t use TOR/VPN, suspicious domain or IP for this attack. Though the attacker can use it (and if he did), the alert would look like the following for suspicious Domain and IP.

Queries for Global Alerts

Denial of Service:

Within the last 10 minutes, if the number of DNS requests exceeds 50,000 in our cluster. This number may vary according to the size of your organisation.

apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: dns-dos
spec:
  description: "DNS DOS/overload in cluster"
  severity: 80
  lookback: 10m
  period: 10m
  dataSet: dns
  aggregateBy: [client_namespace, client_name_aggr, client_name]
  field: count
  metric: sum
  condition: gt
  threshold: 50000

Cloud API attacks:

The following configuration detects any attacks on a cloud API from a default namespace. We can modify the following configuration for cloud etcd, RBAC, and other cloud resources.

apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: metadata-api
spec:
  nets:
  - 169.254.169.254

---

apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.metadata.requests
spec:
  description: "Metadata API: Connection to metadata service from user namespace"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: (dest_name_aggr='metadata-api' OR dest_name_aggr='kse.kubernetes') AND action='allow' AND proto='tcp' AND reporter=src AND (source_namespace='default' OR source_namespace='crown-space')

Earlier attacker made a request to the cloud metadata service from compromised Drupal-pod to determine token privileges and Calico Enterprise Global Alerts detected the attempts as follows:

Summary:

During each step of the attack, we saw that Calico Enterprise Global Alerts was able to detect the attack and produce a high-value alert with information that enables your SOC team to readily analyze the threat and thwart the attacker.

When properly configured to monitor your Kubernetes workloads in cloud environments, Calico Enterprise Global Alerts is highly effective at detecting unauthorized network activity and malicious behavior. Even a single connection can reveal an attacker’s activity.

With these examples, we have also shown how Calico Enterprise Global Alerts can be used to configure threat feeds for IP and domains, and to detect denial of service (DOS) was well as cloud metadata attacks.

————————————————-

Free Online Training
Access Live and On-Demand Kubernetes Tutorials

Calico Enterprise – Free Trial
Solve Common Kubernetes Roadblocks and Advance Your Enterprise Adoption

Join our mailing list

Get updates on blog posts, workshops, certification programs, new releases, and more!

X