Organizations are rapidly moving more and more mission-critical applications to Kubernetes (K8s) and the cloud to reduce costs, achieve faster deployment times, and improve operational efficiencies, but are struggling to achieve a strong security posture because of their inability to apply conventional security practices in the cloud environment. Commitment to cloud security grows, but security safeguards are not keeping up with the increased use of the various cloud platforms. Regardless of the cloud provider or service model, individual organizations are ultimately responsible for the security of their data.
According to a 2019 Ponemon Institute Global Cloud Data Security Study, 70 percent of respondents find it more complex to manage privacy and data protection regulations in a cloud environment than on-premises. Meanwhile, the percent of corporate data stored in the cloud environment has grown from an average of 30 percent in 2015 to an average of 48 percent in 2019. In the same study, 56 percent of respondents say the use of cloud resources increases compliance risk.
The downside associated with a security breach is severe for any organization, but especially so for companies in regulated environments like financial services, healthcare and telecommunications. Now there’s a new and highly effective way for security operations teams responsible for protecting Kubernetes workloads to gain greater control of their cloud security posture while reducing operational complexity.
Using Calico Enterprise with Global Alerts, a new feature just released, organizations can easily monitor critical Kubernetes workloads in any cloud environment, quickly identify malicious behavior, and significantly reduce dwell time.
Calico Enterprise with Global Alerts provides advanced threat and lateral movement detection capabilities for Kubernetes workloads in the cloud, enabling you to more easily uncover malicious activity in advanced stages.
In this blog we will show you how an attacker compromises a public-facing application and gains a foothold in a cloud network. Then he goes on to find his next target within the cluster, moving laterally to further his attack campaign. At the same time we will show you how Global Alerts, a new feature within Calico Enterprise, detects these malicious activities at each stage.
We will launch a cloud infrastructure attack by compromising multiple pods and move laterally into the cloud infrastructure, taking advantage of any weaknesses. Our front-end website is using Drupal application running on Drupal-pod in default namespace, on port 80. And Webmin service is running under namespace ‘crown-space’ which is not exposed to the internet. Webmin controls and monitors server resources and is accessible on port 10000.
We will configure the Global Alerts around these critical apps so that monitoring of these resources is automated.
First, let’s review aspects of our scenario which may uncover an attacker’s behavior.
In the next step, we configure Global alerts to detect and respond to the above conditions.
Let’s configure GlobalAlert using following manifests.
apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.lateral.from-drupal spec: description: "Lateral movement detected: Outgoing Connection from Drupal" severity: 100 period: 10m lookback: 10m dataSet: flows query: "source_name_aggr='drupal-pod' AND proto='tcp' AND action='allow' AND reporter=src AND NOT (source_port=80) AND NOT (dest_name_aggr='kse.kubernetes') AND NOT (dest_name_aggr='pub') AND NOT (dest_port=10000) AND NOT (dest_name_aggr='metadata-api')"
apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.lateral.to-webmin spec: description: "Lateral movement detected: Webmin accessed" severity: 100 period: 10m lookback: 10m dataSet: flows query: "dest_name_aggr='webmin-pod' AND proto='tcp' AND action='allow' AND reporter='src' AND NOT (dest_port=10000)"
apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.lateral.from-webmin spec: description: "Lateral movement detected: Outgoing connection from Webmin" severity: 100 period: 10m lookback: 10m dataSet: flows query: "source_name_aggr='webmin-pod' AND proto='tcp' AND action='allow' AND reporter=src AND NOT (source_port=10000) AND NOT (dest_name_aggr='kse.kubernetes') AND NOT (dest_name_aggr='pub') AND NOT (dest_name_aggr='metadata-api')"
apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.metadata.requests spec: description: "Metadata API: Connection to metadata service from user namespace" severity: 100 period: 10m lookback: 10m dataSet: flows query: (dest_name_aggr='metadata-api' OR dest_name_aggr='kse.kubernetes') AND action='allow' AND proto='tcp' AND reporter=src AND (source_namespace='default' OR source_namespace='crown-space')
Configure Global Network Set for metadata API.
apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: metadata-api spec: nets: - 169.254.169.254
Suspicious Domains threat feed.
#manifest 6 apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: global.threat.domains spec: content: DomainNameSet pull: http: url: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt
#manifest 7 apiVersion: projectcalico.org/v3 kind: GlobalThreatFeed metadata: name: global.threat.ipfeodo spec: pull: http: url: https://feodotracker.abuse.ch/downloads/ipblocklist.txt globalNetworkSet: labels: feed: feodo
Suspicious IP feed, block policy.
apiVersion: projectcalico.org/v3 kind: GlobalNetworkPolicy metadata: name: default.block-feodo spec: tier: default selector: app == 'crown-webmin' types: - Egress egress: - action: Deny destination: selector: feed == 'feodo' - action: Allow
We recommend that you review the Global Alerts documentation to understand the various fields, some of which we will discuss. All manifests run every 10 minutes and look at data T-10 minutes. (period and lookback)
Let’s look at a front end application on the cloud. The public drupal site is accessible on port 80 and serves the following page.
To find the version of Drupal, we will connect to the site to look at HTTP headers. The banner grab, below, indicates the installation is Drupal 8.
Drupal 8 has a ‘Form property injection’ vulnerability, i.e. CVE-2018-7600, and has an exploit module with metasploit ‘unix/webapp/drupalgeddon2’.
The exploit targets the Drupal form fields API. Unvalidated form fields on the page are used to inject malicious array into form API so that drupal AJAX callback renders array by executing it.
By running the metasploit exploit on the Drupal server, you can see a reverse shell is spawned exploiting the vulnerability successfully on port 4444. We see a Drupal daemon process is running as non-root. Though we can now copy backdooring tools like netcat and kubectl binaries on Drupal-pod.
./kubectl auth can-i --list Resources Non-Resource URLs Resource Names Verbs globalnetworkpolicies.projectcalico.org   [*] networkpolicies.projectcalico.org   [*] stagedglobalnetworkpolicies.projectcalico.org   [*] stagednetworkpolicies.projectcalico.org   [*] selfsubjectaccessreviews.authorization.k8s.io   [create] selfsubjectrulesreviews.authorization.k8s.io   [create] [/api/*]  [get] [/api]  [get] [/apis/*]  [get] [/apis]  [get] [/healthz]  [get] [/openapi/*]  [get] [/openapi]  [get] [/swagger-2.0.0.pb-v1]  [get] [/swagger.json]  [get] [/swaggerapi/*]  [get] [/swaggerapi]  [get] [/version/]  [get] [/version]  [get]
We have three ways to proceed:
Look at the following picture. This is the alert that was generated, when Drupal was compromised and the reverse shell was spawned.
Once we have control of the Drupal pod, with the lack of privilege and access, we look into other exposed resources that are reachable from this pod. By enumerating the network, we see there is a webmin application running on port 10000.
Now, the HTTP banner grab confirms MiniServ 1.910 is running. The attacker now has an interesting target as this version of MiniServ is backdoored which is identified by CVE-2019-15107.
The backdoor was hidden for over a year and eventually discovered. The exploit targets the password change function where ‘old password’ is executed as command. Metasploit has a module unix/webapp/webmin_backdoor which exploits this vulnerability.
Now, to reach the Webmin application from the attacker machine, the compromised Drupal-pod can be used as a network relay.
The following command will create a relay between the attacker and Drupal-pod so that the attacker’s requests will be forwarded to the Webmin application.
We can reach the Webmin application by making requests to port 5000 locally from the attacker machine as follows.
attacker# curl -kv https://127.0.0.1:5000
Now we can send an exploit to Webmin through Drupal-pod and we will listen for the reverse shell on Drupal-pod on port 2222. Why? There is no way for the Webmin-application to talk to the attacker directly as it’s an internal app (which we consider as East-West traffic).
Let’s set our metasploit variables RHOST as 127.0.0.0 and RPORT as 5000 (relay to Webmin-application) and LHOST as Drupal-pod IP on attacker metasploit.
And on Drupal-pod, we want to make sure we are listening on port 2222 with the copied netcat binary.
Note that the handler fails to bind on the attacker machine as our LHOST is set as rupal-pod’s IP. But we successfully get the shell on Drupal-pod where we are listening on port 2222. That means the exploit was successfully sent to the Webmin-application.
Now we have root access on the Webmin-pod (shown in the image above) with uid=0. With root access, the attacker can use this pod to make changes to the file system and cover his tracks by deleting logs. The attacker can also install malware, miners, and stealers and also use it as a pivot to launch further attacks.
Reverse shell from Webmin to Drupal :
Drupal to Webmin attack connection:
In this example, I didn’t use TOR/VPN, suspicious domain or IP for this attack. Though the attacker can use it (and if he did), the alert would look like the following for suspicious Domain and IP.
Denial of Service:
Within the last 10 minutes, if the number of DNS requests exceeds 50,000 in our cluster. This number may vary according to the size of your organisation.
apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: dns-dos spec: description: "DNS DOS/overload in cluster" severity: 80 lookback: 10m period: 10m dataSet: dns aggregateBy: [client_namespace, client_name_aggr, client_name] field: count metric: sum condition: gt threshold: 50000
Cloud API attacks:
The following configuration detects any attacks on a cloud API from a default namespace. We can modify the following configuration for cloud etcd, RBAC, and other cloud resources.
apiVersion: projectcalico.org/v3 kind: GlobalNetworkSet metadata: name: metadata-api spec: nets: - 169.254.169.254 --- apiVersion: projectcalico.org/v3 kind: GlobalAlert metadata: name: global.metadata.requests spec: description: "Metadata API: Connection to metadata service from user namespace" severity: 100 period: 10m lookback: 10m dataSet: flows query: (dest_name_aggr='metadata-api' OR dest_name_aggr='kse.kubernetes') AND action='allow' AND proto='tcp' AND reporter=src AND (source_namespace='default' OR source_namespace='crown-space')
Earlier attacker made a request to the cloud metadata service from compromised Drupal-pod to determine token privileges and Calico Enterprise Global Alerts detected the attempts as follows:
During each step of the attack, we saw that Calico Enterprise Global Alerts was able to detect the attack and produce a high-value alert with information that enables your SOC team to readily analyze the threat and thwart the attacker.
When properly configured to monitor your Kubernetes workloads in cloud environments, Calico Enterprise Global Alerts is highly effective at detecting unauthorized network activity and malicious behavior. Even a single connection can reveal an attacker’s activity.
With these examples, we have also shown how Calico Enterprise Global Alerts can be used to configure threat feeds for IP and domains, and to detect denial of service (DOS) was well as cloud metadata attacks.
Free Online Training
Access Live and On-Demand Kubernetes Tutorials
Calico Enterprise – Free Trial
Solve Common Kubernetes Roadblocks and Advance Your Enterprise Adoption
Get updates on blog posts, new releases and more!