--- title: "Five Ways to Quickly Uncover Malicious Activity and Protect Your Kubernetes Workloads" source: "https://www.tigera.io/blog/five-ways-to-quickly-uncover-malicious-activity-and-protect-your-kubernetes-workloads/" --- [Technical Blog](https://www.tigera.io/category/technical-blog/) # Five Ways to Quickly Uncover Malicious Activity and Protect Your Kubernetes Workloads By [Manoj Ahuje](https://www.tigera.io/blog/author/manoj-ahuje/) on Dec 12, 2019 • 13 min read Organizations are rapidly moving more and more mission-critical applications to Kubernetes (K8s) and the cloud to reduce costs, achieve faster deployment times, and improve operational efficiencies, but are struggling to achieve a strong security posture because of their inability to apply conventional security practices in the cloud environment. Commitment to cloud security grows, but security safeguards are not keeping up with the increased use of the various cloud platforms. Regardless of the cloud provider or service model, individual organizations are ultimately responsible for the security of their data. ## Security breach stats According to a 2019 Ponemon Institute Global Cloud Data Security Study, 70 percent of respondents find it more complex to manage privacy and data protection regulations in a cloud environment than on-premises. Meanwhile, the percent of corporate data stored in the cloud environment has grown from an average of 30 percent in 2015 to an average of 48 percent in 2019. In the same study, 56 percent of respondents say the use of cloud resources increases compliance risk. The downside associated with a security breach is severe for any organization, but especially so for companies in regulated environments like financial services, healthcare and telecommunications. Now there’s a new and highly effective way for security operations teams responsible for protecting Kubernetes workloads to gain greater control of their cloud security posture while reducing operational complexity. Using Calico Enterprise with Global Alerts, a new feature just released, organizations can easily monitor critical Kubernetes workloads in any cloud environment, quickly identify malicious behavior, and significantly reduce dwell time. ## Calico Enterprise can be used to detect: - Lateral movement by the attacker for East-West and North-South traffic - You can monitor all flows within your cluster for threat and lateral movement detection - Botnet, Command and Control, VPN-TOR - You can integrate intelligence feeds with Calico Enterprise using Global Alerts threat feeds - Malicious DNS queries - Using suspicious domain feeds, Calico Enterprise with Global Alerts flags any DNS requests to a malicious domain from your workload - Cloud Metadata API attacks. - You can identify workload flows that attempt to connect to cloud resources like metadata API if you are on Kubernetes etcd, RBAC, etc. where an attacker attempts to enumerate access and elevate his/her privileges - DGA malware traffic. - Using a machine learning algorithm, you can detect any malicious DGA domain traffic from your cluster. We’ll discuss this in more detail in a future blog Calico Enterprise with Global Alerts provides advanced threat and lateral movement detection capabilities for Kubernetes workloads in the cloud, enabling you to more easily uncover malicious activity in advanced stages. In this blog we will show you how an attacker compromises a public-facing application and gains a foothold in a cloud network. Then he goes on to find his next target within the cluster, moving laterally to further his attack campaign. At the same time we will show you how Global Alerts, a new feature within Calico Enterprise, detects these malicious activities at each stage. ## Attack scenario: We will launch a cloud infrastructure attack by compromising multiple pods and move laterally into the cloud infrastructure, taking advantage of any weaknesses. Our front-end website is using Drupal application running on Drupal-pod in default namespace, on port 80. And Webmin service is running under namespace ‘crown-space’ which is not exposed to the internet. Webmin controls and monitors server resources and is accessible on port 10000. We will configure the Global Alerts around these critical apps so that monitoring of these resources is automated. First, let’s review aspects of our scenario which may uncover an attacker’s behavior. - Lateral movement attempt from Drupal-pod - Any connection originating from Drupal-pod to internal network apart from approved flows - Any connection made to Drupal-pod apart from its service port 80 - Lateral movement attempt to Webmin-pod - Any connection made to Webmin-pod apart from its service port 10000 - Any connection originating from webmin-pod apart from approved flow - Cloud metadata API requests from crown-space and default namespace - To detect user-made cloud metadata API requests which are unusual in a Kubernetes pod - Threat feeds - Monitor any connection made to suspicious IP or domains from the Kubernetes cluster - Attacker or one of his pivoting tools may end up talking to suspicious IP or domain In the next step, we configure Global alerts to detect and respond to the above conditions. ## Configure GlobalAlerts : Let’s configure GlobalAlert using following manifests. ### Manifest 1: - Checks if there is any connection originating from Drupal-pod - We want to whitelist any approved flow - For Drupal it is port 80 - Approved flows can be added with ‘NOT’ operator in query ``` `apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.lateral.from-drupal spec: description: "Lateral movement detected: Outgoing Connection from Drupal" severity: 100 period: 10m lookback: 10m dataSet: flows query: "source_name_aggr='drupal-pod' AND proto='tcp' AND action='allow' AND reporter=src AND NOT (source_port=80) AND NOT (dest_name_aggr='kse.kubernetes') AND NOT (dest_name_aggr='pub') AND NOT (dest_port=10000) AND NOT (dest_name_aggr='metadata-api')"` ``` ### Manifest 2: - Check if the Webmin-pod is accessed from any other port apart from 10000 - This condition indicates an unapproved open port which could be bind shell on webmin pod ``` `apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.lateral.to-webmin spec: description: "Lateral movement detected: Webmin accessed" severity: 100 period: 10m lookback: 10m dataSet: flows query: "dest_name_aggr='webmin-pod' AND proto='tcp' AND action='allow' AND reporter='src' AND NOT (dest_port=10000)"` ``` ### Manifest 3: - Any connection originating from webmin-pod apart from port 10000 - This condition indicates a reverse shell or unapproved activity - We can use ‘NOT’ operator to exclude approved flows ``` `apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.lateral.from-webmin spec: description: "Lateral movement detected: Outgoing connection from Webmin" severity: 100 period: 10m lookback: 10m dataSet: flows query: "source_name_aggr='webmin-pod' AND proto='tcp' AND action='allow' AND reporter=src AND NOT (source_port=10000) AND NOT (dest_name_aggr='kse.kubernetes') AND NOT (dest_name_aggr='pub') AND NOT (dest_name_aggr='metadata-api')"` ``` ### Manifest 4: - Any requests made to cloud provider’s metadata API from application namespace - This is very suspicious within a cloud environment if your namespace isn’t supposed to use this API for any legitimate activity ``` `apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.metadata.requests spec: description: "Metadata API: Connection to metadata service from user namespace" severity: 100 period: 10m lookback: 10m dataSet: flows query: (dest_name_aggr='metadata-api' OR dest_name_aggr='kse.kubernetes') AND action='allow' AND proto='tcp' AND reporter=src AND (source_namespace='default' OR source_namespace='crown-space')` ``` ### Manifest 5: Configure Global Network Set for metadata API. ``` `apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: metadata-api spec: nets: - 169.254.169.254` ``` ### Manifest 5a: Suspicious Domains threat feed. ``` `#manifest 6 apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: global.threat.domains spec: content: DomainNameSet pull: http: url: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt` ``` ### Manifest 5b: Suspicious IP Threat feed. You can find tor exit node feed [here](https://check.torproject.org/cgi-bin/TorBulkExitList.py?ip=1.1.1.1) and ejr-vpn feed here. These feeds can be used under pull method. ``` `#manifest 7 apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: global.threat.ipfeodo spec: pull: http: url: http://feodotracker.abuse.ch/downloads/ipblocklist.txt globalNetworkSet: labels: feed: feodo` ``` ### Manifest 5c: Suspicious IP feed, block policy. ``` `apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.block-feodo spec: tier: default selector: app == 'crown-webmin' types: - Egress egress: - action: Deny destination: selector: feed == 'feodo' - action: Allow` ``` We recommend that you review the  Global Alerts documentation to understand the various [fields](https://docs.tigera.io/v2.6/reference/resources/globalalert#globalalert-definition), some of which we will discuss. All manifests run every 10 minutes and look at data T-10 minutes. (period and lookback) ![Diagram of Internet traffic flowing through pods to Drupal and Webmin applications](/app/uploads/2019/12/Global-Alert-diagram.png)   ## Attack on Drupal Site: Let’s look at a front end application on the cloud. The public drupal site is accessible on port 80 and serves the following page. ![Drupal site served on port 80. Front page with "Welcome to mysite" and a link to the User Guide](/app/uploads/2019/12/Global-Alert-Blog12.jpg) To find the version of Drupal, we will connect to the site to look at HTTP headers. The banner grab, below, indicates the installation is Drupal 8. ![HTTP headers showing "X-Generator: Drupal 8," indicating the site is running Drupal 8](/app/uploads/2019/12/Global-Alert-Blog07.jpg) Drupal 8 has a ‘Form property injection’ vulnerability, i.e. CVE-2018-7600, and has an exploit module with metasploit ‘unix/webapp/drupalgeddon2’. The exploit targets the Drupal form fields API. Unvalidated form fields on the page are used to inject malicious array into form API so that drupal AJAX callback renders array by executing it. By running the metasploit exploit on the Drupal server, you can see a reverse shell is spawned exploiting the vulnerability successfully on port 4444. We see a Drupal daemon process is running as non-root. Though we can now copy backdooring tools like netcat and kubectl binaries on Drupal-pod. ![Metasploit console showing successful Drupal exploit, reverse shell spawned on port 4444, Meterpreter session opened](/app/uploads/2019/12/Global-Alert-Blog11.jpg) ![Terminal output showing a meterpreter shell with "www-data" user ID](/app/uploads/2019/12/Global-Alert-Blog10.jpg) ``` `./kubectl auth can-i --list Resources Non-Resource URLs Resource Names Verbs globalnetworkpolicies.projectcalico.org [] [] [*] networkpolicies.projectcalico.org [] [] [*] stagedglobalnetworkpolicies.projectcalico.org [] [] [*] stagednetworkpolicies.projectcalico.org [] [] [*] selfsubjectaccessreviews.authorization.k8s.io [] [] [create] selfsubjectrulesreviews.authorization.k8s.io [] [] [create] [/api/*] [] [get] [/api] [] [get] [/apis/*] [] [get] [/apis] [] [get] [/healthz] [] [get] [/openapi/*] [] [get] [/openapi] [] [get] [/swagger-2.0.0.pb-v1] [] [get] [/swagger.json] [] [get] [/swaggerapi/*] [] [get] [/swaggerapi] [] [get] [/version/] [] [get] [/version] [] [get]` ```   We have three ways to proceed: - Try to get root privileges at drupal-pod (which is not always possible) or - Abuse service token in our case it doesn’t have special priviledges - Find another target within the cluster and move laterally to gain greater privileges. We will see the later part of the attack in the next section. **Detection:** - Check Global Alerts on Calico Enterprise. As soon as your felix interval is finished, you will see an alert with details of the connection, above - Use these details to track the flow in Elasticsearch and traffic at that time - We can see the global.lateral.from-drupal alert being triggered - At this stage an incident response team can be notified to investigate and uncover the compromise Look at the following picture. This is the alert that was generated, when Drupal was compromised and the reverse shell was spawned. ![Screenshot of a security alert: "Lateral movement detected: Outgoing Connection from Drupal." JSON data shows network flow](/app/uploads/2019/12/Global-Alert-Blog09.jpg)   ## Attack on Webmin server: Once we have control of the Drupal pod, with the lack of privilege and access, we look into other exposed resources that are reachable from this pod. By enumerating the network, we see there is a webmin application running on port 10000. Now, the HTTP banner grab confirms MiniServ 1.910 is running. The attacker now has an interesting target as this version of MiniServ is backdoored which is identified by CVE-2019-15107. The backdoor was hidden for over a year and eventually discovered. The exploit targets the password change function where ‘old password’ is executed as command. Metasploit has a module unix/webapp/webmin_backdoor which exploits this vulnerability. ![HTTP response headers from MiniServ/1.910, indicating the vulnerable Webmin server targeted by the Metasploit module](/app/uploads/2019/12/Global-Alert-Blog06.jpg) Now, to reach the Webmin application from the attacker machine, the compromised Drupal-pod can be used as a network relay. The following command will create a relay between the attacker and Drupal-pod so that the attacker’s requests will be forwarded to the Webmin application. ![Command line output showing the creation of a TCP relay from port 5000 to 192.168.89.99:10000](/app/uploads/2019/12/Global-Alert-Blog05.jpg) We can reach the Webmin application by making requests to port 5000 locally from the attacker machine as follows. ``` `attacker# curl -kv http://127.0.0.1:5000` ```   ![Webmin login screen at 127.0.0.1, suggesting successful connection via `curl` command](/app/uploads/2019/12/Global-Alert-Blog.jpg) Now we can send an exploit to Webmin through Drupal-pod and we will listen for the reverse shell on Drupal-pod on port 2222. Why? There is no way for the Webmin-application to talk to the attacker directly as it’s an internal app (which we consider as East-West traffic). Let’s set our metasploit variables RHOST as 127.0.0.0 and RPORT as 5000 (relay to Webmin-application) and LHOST as Drupal-pod IP on attacker metasploit. And on Drupal-pod, we want to make sure we are listening on port 2222 with the copied netcat binary. ![Terminal output showing netcat listening on port 2222 on the Drupal-pod, confirming successful setup](/app/uploads/2019/12/Global-Alert-Blog02.jpg) Note that the handler fails to bind on the attacker machine as our LHOST is set as rupal-pod’s IP. But we successfully get the shell on Drupal-pod where we are listening on port 2222. That means the exploit was successfully sent to the Webmin-application. Now we have root access on the Webmin-pod (shown in the image above) with uid=0. With root access, the attacker can use this pod to make changes to the file system and cover his tracks by deleting logs. The attacker can also install malware, miners, and stealers and also use it as a pivot to launch further attacks. **Detection:** - Check the Global Alerts tab on Calico Enterprise to view this lateral move from Drupal-pod to Webmin-pod - We will see Manifests 2 and 3 are triggered - We can further investigate connections, traffic, and movement at the same time using inputs from the alert Reverse shell from Webmin to Drupal : ![Network log showing a lateral movement alert: outgoing connection from Webmin, possibly indicating a reverse shell to Drupal](/app/uploads/2019/12/Global-Alert-Blog03.jpg)   Drupal to Webmin attack connection: ![Screenshot of network traffic analysis showing an outgoing connection from Drupal to a "crown-space" destination](/app/uploads/2019/12/Global-Alert-Blog04.jpg) ## Additional Alerts ## ThreatFeeds: In this example, I didn’t use TOR/VPN, suspicious domain or IP for this attack. Though the attacker can use it (and if he did), the alert would look like the following for suspicious Domain and IP. ## ![Alert details showing a suspicious domain query, pagaldaily.com, by attacker-pod without using TOR/VPN. Source IP is N/A](https://www.tigera.io/app/uploads/2019/12/lightshot.png) ## Queries for Global Alerts **Denial of Service:** Within the last 10 minutes, if the number of DNS requests exceeds 50,000 in our cluster. This number may vary according to the size of your organisation. ``` `apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: dns-dos spec: description: "DNS DOS/overload in cluster" severity: 80 lookback: 10m period: 10m dataSet: dns aggregateBy: [client_namespace, client_name_aggr, client_name] field: count metric: sum condition: gt threshold: 50000` ``` **Cloud API attacks:** The following configuration detects any attacks on a cloud API from a default namespace. We can modify the following configuration for cloud etcd, RBAC, and other cloud resources. ``` `apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: metadata-api spec: nets: - 169.254.169.254 --- apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.metadata.requests spec: description: "Metadata API: Connection to metadata service from user namespace" severity: 100 period: 10m lookback: 10m dataSet: flows query: (dest_name_aggr='metadata-api' OR dest_name_aggr='kse.kubernetes') AND action='allow' AND proto='tcp' AND reporter=src AND (source_namespace='default' OR source_namespace='crown-space')` ``` Earlier attacker made a request to the cloud metadata service from compromised Drupal-pod to determine token privileges and Calico Enterprise Global Alerts detected the attempts as follows: ![Calico Enterprise Global Alert showing metadata API connection from Drupal pod, including source labels "app=drupal](/app/uploads/2019/12/Global-Alert-Blog08.jpg) ## Summary: During each step of the attack, we saw that Calico Enterprise Global Alerts was able to detect the attack and produce a high-value alert with information that enables your SOC team to readily analyze the threat and thwart the attacker. When properly configured to monitor your Kubernetes workloads in cloud environments, Calico Enterprise Global Alerts is highly effective at detecting unauthorized network activity and malicious behavior. Even a single connection can reveal an attacker’s activity. With these examples, we have also shown how Calico Enterprise Global Alerts can be used to configure threat feeds for IP and domains, and to detect denial of service (DOS) was well as cloud metadata attacks. ————————————————- [**Free Online Training**](https://www.tigera.io/events/) Access Live and On-Demand Kubernetes Tutorials [**Calico Enterprise – Free Trial**](https://www.calicocloud.io/home) Solve Common Kubernetes Roadblocks and Advance Your Enterprise Adoption [How-To](https://www.tigera.io/tags/how-to/)[Products](https://www.tigera.io/tags/products/) ## Related posts [![Meet Mylo: An AI-native way to work with Calico](https://www.tigera.io/app/uploads/2026/09/Meet-Mylo-An-AI-native-way-to-work-with-Calico.png)](https://www.tigera.io/blog/meet-mylo-an-ai-native-way-to-work-with-calico/) #### [Meet Mylo: An AI-native way to work with Calico](https://www.tigera.io/blog/meet-mylo-an-ai-native-way-to-work-with-calico/) By [Phil DiCorpo](https://www.tigera.io/blog/author/phil-dicorpo/) on Sep 3, 2026 A library of Calico tools and skills — delivered through the Calico MCP Server What if your hardest network question took ten minutes instead of ten days? Anyone who has operated Kubernetes networking at scale... [Read more](https://www.tigera.io/blog/meet-mylo-an-ai-native-way-to-work-with-calico/) [![The Safest Place to Run an AI Agent Is On a Cluster That Doesn’t Trust It](https://www.tigera.io/app/uploads/2026/08/The-Safest-Place-to-Run-an-AI-Agent-Is-On-a-Cluster-That-Doesnt-Trust-It.png)](https://www.tigera.io/blog/the-safest-place-to-run-an-ai-agent-is-on-a-cluster-that-doesnt-trust-it/) #### [The Safest Place to Run an AI Agent Is On a Cluster That Doesn’t Trust It](https://www.tigera.io/blog/the-safest-place-to-run-an-ai-agent-is-on-a-cluster-that-doesnt-trust-it/) By [Alister Baroi](https://www.tigera.io/blog/author/alister-baroi/) on Aug 27, 2026 Every organization running AI agents has already made a hosting decision. Most made it by accident. The sales team switched on the agent built into their CRM. Engineering is piloting a coding agent in a... [Read more](https://www.tigera.io/blog/the-safest-place-to-run-an-ai-agent-is-on-a-cluster-that-doesnt-trust-it/) [![AI Red Team Agents Automate Attacks on your AI Agents. Runtime Policies Automate their Defense.](https://www.tigera.io/app/uploads/2026/08/AI-Red-Team-Agents-Automate-Attacks-on-your-AI-Agents.-Runtime-Policies-Automate-their-Defense.png)](https://www.tigera.io/blog/ai-red-team-agents-automate-attacks-on-your-ai-agents-runtime-policies-automate-their-defense/) #### [AI Red Team Agents Automate Attacks on your AI Agents. Runtime Policies Automate their Defense.](https://www.tigera.io/blog/ai-red-team-agents-automate-attacks-on-your-ai-agents-runtime-policies-automate-their-defense/) By [Alister Baroi](https://www.tigera.io/blog/author/alister-baroi/) on Aug 24, 2026 The AI red teaming market grew up fast this year. OpenAI bought Promptfoo, Cisco and Microsoft shipped automated attack suites, and a seed-stage startup publicly compromised 50 of 55 live customer service bots. These platforms... [Read more](https://www.tigera.io/blog/ai-red-team-agents-automate-attacks-on-your-ai-agents-runtime-policies-automate-their-defense/)