Detect malicious activity and protect your containerized workloads in Amazon EKS or AWS

As containerized applications become increasingly complex, it can be challenging to design and execute an effective container security strategy. With the growing trend towards cloud-based applications and services, cyber criminals are also evolving their attack techniques, making container security solutions more critical than ever. Calico provides robust detection capabilities to detect known and zero-day container and network-based attacks. In this blog, we will look at Calico’s capabilities to detect network-based attacks.

Calico offers comprehensive protection against both known and zero-day network-based attacks. Using a combination of workload-based IDS/IPS, Calio can detect and block connections to known malicious IPs identified with AlienVault and custom threat intelligence feeds. Calico also uses heuristics-based learning to identify anomalous network activity and prevent zero-day attacks. To further protect against OWASP Top 10 attacks, Calico provides a web application firewall (WAF) that can intercept attacks and prevent them from reaching your applications. Additionally, Calico can also block requests from malicious IPs to prevent DDoS attacks from overwhelming your system.

Malicious attack on cloud-native application and how to prevent it

In this blog, we will go through a scenario where an attacker compromises a public-facing application and gains a foothold in the AWS EC2 or EKS network of this application. Then he goes on to find his next target within the Kubernetes cluster, moving laterally to further his attack campaign. In parallel, let’s leverage Globalthreats with Calico on AWS and EKS to detect these malicious activities at each stage.

Our front-end website is using the Drupal application running on Drupal-pod in the default namespace, on port 80. And Webmin service is running under namespace ‘crown-space’ which is not exposed to the internet. Webmin controls and monitors server resources and is accessible on port 10000.

Step 1: Configure the Global Alerts around these critical apps to monitor these resources automatically. You can uncover an attacker’s behavior by observing the following:

Step 2: You can apply the following manifests as you deploy the application (Figure 1) to protect for above observations

Lateral movement attempt from Drupal-pod

Let’s look at a front-end application on the cloud. The public drupal site is accessible on port 80 and serves the webpage in Figure 2.

To find the version of Drupal, let’s connect to the site to look at HTTP headers. The banner grab, in Figure 4 indicates the installation is Drupal 8.

Drupal 8 has a ‘Form property injection’ vulnerability, i.e. CVE-2018-7600, and has an exploit module with metasploit ‘unix/webapp/drupal_drupalgeddon2’.
The exploit targets the Drupal form fields API. Unvalidated form fields on the page are used to inject malicious array into the form API so that drupal AJAX callback renders array by executing it.
By running the metasploit exploit on the Drupal server, you can see a reverse shell is spawned exploiting the vulnerability successfully on port 4444 in Figure 5.

You will notice that a Drupal daemon process is running as non-root but backdooring tools like netcat and kubectl binaries can be copied on Drupal-pod as seen in Figure 6.

./kubectl auth can-i --list
Resources                                       Non-Resource URLs        Resource Names   Verbs
globalnetworkpolicies.projectcalico.org         []                       []               [*]
networkpolicies.projectcalico.org               []                       []               [*]
stagedglobalnetworkpolicies.projectcalico.org   []                       []               [*]
stagednetworkpolicies.projectcalico.org         []                       []               [*]
selfsubjectaccessreviews.authorization.k8s.io   []                       []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                       []               [create]
                                                [/api/*]                 []               [get]
                                                [/api]                   []               [get]
                                                [/apis/*]                []               [get]
                                                [/apis]                  []               [get]
                                                [/healthz]               []               [get]
                                                [/openapi/*]             []               [get]
                                                [/openapi]               []               [get]
                                                [/swagger-2.0.0.pb-v1]   []               [get]
                                                [/swagger.json]          []               [get]
                                                [/swaggerapi/*]          []               [get]
                                                [/swaggerapi]            []               [get]
                                                [/version/]              []               [get]
                                                [/version]               []               [get]

There are three ways to proceed:

1. Try to get root privileges at drupal-pod (which is not always possible) or
2. Abuse service token in this case as it doesn’t have special privileges
3. Find another target within the cluster and move laterally to gain greater privileges.
   1. We will see the later part of the attack in the next section.

Detection and Prevention

Use following yaml to do following:

• Check if there are any connections originating from Drupal-pod
• Whitelist any approved flow
• Add approved flows with ‘NOT’ operator in query

Manifest 1

apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.lateral.from-drupal
spec:
  description: "Lateral movement detected: Outgoing Connection from Drupal"
  summary: "[flows] [lateral movement] ${source_namespace}/${source_name_aggr} with label app=drupal initiated connection"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: '"source_labels.labels"="app=drupal" AND proto="tcp" AND action="allow" AND reporter="src"'
  aggregateBy: [source_namespace, source_name_aggr]
  field: num_flows
  metric: sum
  condition: gt
  threshold: 0

Results

As soon as your felix interval is finished, you will see an alert with details of the reverse connection from drupal (Figure 7). You can take following steps:

• Use these details to track the flow in Elasticsearch and traffic at that time
• Observe the global.lateral.from-drupal alert being triggered
• Notify the incident response team to investigate and uncover the compromise

Example Manifest1 after adjusting for allowed connections where “app: drupal-mysql” is a label on :

apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.lateral.from-drupal
spec:
  description: "Lateral movement detected: Outgoing Connection from Drupal"
  summary: "[flows] [lateral movement] ${source_namespace}/${source_name_aggr} with label app=drupal initiated connection"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: '"source_labels.labels"="app=drupal" AND proto="tcp" AND action="allow" AND reporter="src" AND "dest_labels.labels"!="app=drupal-mysql"'
  aggregateBy: [source_namespace, source_name_aggr]
  field: num_flows
  metric: sum
  condition: gt
  threshold: 0

Lateral movement attempt to Webmin-pod:

Once an attacker has control of the Drupal pod, with the lack of privilege and access, they will look into other exposed resources that are reachable from this pod. By enumerating the network, you can see there is a Webmin application running on port 10000 in Figure 8.

Now, the HTTP banner grab confirms MiniServ 1.910 is running. The attacker now has an interesting target as this version of MiniServ is backdoored which is identified by CVE-2019-15107.

Now, to reach the Webmin application from the attacker machine, the compromised Drupal-pod can be used as a network relay. The following command will create a relay between the attacker and Drupal-pod so that the attacker’s requests will be forwarded to the Webmin application.

The attacker can reach the Webmin application by making requests to port 5000 locally from his machine as follows.

attacker# curl -kv https://127.0.0.1:5000

Now an exploit to Webmin through Drupal-pod can be sent and listen for the reverse shell on Drupal-pod on port 2222. Why? There is no way for the Webmin-application to talk to the attacker directly as it’s an internal app and this is categorized as East-West traffic.

Let’s set the metasploit variables RHOST as 127.0.0.0 and RPORT as 5000 (relay to Webmin-application) and LHOST as Drupal-pod IP on attacker metasploit.

And on Drupal-pod, listen on port 2222 with the copied netcat binary.

Note that the handler fails to bind on the attacker machine as the LHOST is set as drupal-pod’s IP. But there is a shell on Drupal-pod listening on port 2222. This implies that the exploit was successfully sent to the Webmin-application.

The attacker now has root access on the Webmin-pod (shown in the image above) with uid=0. With root access, the attacker can use this pod to make changes to the file system and cover his tracks by deleting logs. The attacker can also install malware, miners, and stealers and also use it as a pivot to launch further attacks.

Detection and Prevention

Use following yaml

• Check if the Webmin-pod is accessed on non designated port i.e. apart from 10000
• Whitelist any approved flow
• Add approved flows with ‘NOT’ operator in query

Manifest 2

Use the following YAML to check

• Any connection originating from webmin-pod
• A reverse shell or unapproved activity
• Use ‘NOT’ operator to exclude approved flows

Manifest 3

apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.lateral.from-webmin
spec:
  description: "Lateral movement detected: Egress connection from Webmin"
  summary: "[flows] [lateral movement] ${source_namespace}/${source_name_aggr} with label app=webmin made a egress connection"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: '"source_labels.labels"="app=webmin" AND proto="tcp" AND action="allow" AND reporter="src"'
  aggregateBy: [source_namespace, source_name_aggr]
  field: num_flows
  metric: sum
  condition: gt
  threshold: 0

Results

• Check the Global Alerts tab on Calico Enterprise to view this lateral move from Drupal-pod to Webmin-pod
• Observe Manifests 2 and 3 triggered as shown in Figure 9 and 10
• Investigate connections, traffic, and movement at the same time using inputs from the alert in Dynamic Service Graph, Flow visualizer

Reverse shell from Webmin to Drupal:

Drupal to Webmin attack connection:

Additional Alerts

Calico provides a flexible alerting framework that allows security teams to configure various types of alerts targeting information provided in flow logs, dns logs, audit logs, or be triggered as a part of out of the box detection of suspicious behavior.

ThreatFeeds

The attacker can use TOR/VPN, suspicious domain, or IP for an attack and the alert would look like the following for suspicious domain and IP using the following manifest.

• Suspicious Domains threat feed.

Manifest 4a:

apiVersion: projectcalico.org/v3
kind: GlobalThreatFeed
metadata:
  name: global.threat.domains
spec:
  content: DomainNameSet
  pull:
    http:
      url: https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt

• Suspicious IP Threat feed.

Manifest 4b:

apiVersion: projectcalico.org/v3
kind: GlobalThreatFeed
metadata:
  name: global.threat.ipfeodo
spec:
  pull:
    http:
      url: https://feodotracker.abuse.ch/downloads/ipblocklist.txt
  globalNetworkSet:
    labels:
      feed: feodo

---

#Policy to block using above feeds

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: default.block-feodo
spec:
  tier: default
  selector: app == 'webmin'
  types:
  - Egress
  egress:
  - action: Deny
    destination:
      selector: feed == 'feodo'
  - action: Allow

Denial of Service

If the number of DNS requests exceeds the organizational threshold for a time interval, leverage this manifest to be alerted.

apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: dns.dos
spec:
  description: "Alerts when DNS DOS attempt is detected"
  summary: "[dns] DOS attempt detected by ${client_namespace}/${client_name_aggr}"
  severity: 100
  lookback: 10m
  period: 10m
  dataSet: dns
  aggregateBy: [client_namespace, client_name_aggr]
  field: count
  metric: sum
  condition: gt
  threshold: 50000

Cluster state change – Globalnetworkpolicy

If network policy is modified by service account

Manifest 5:

kind: GlobalAlert
metadata:
  name: policy.globalnetworkpolicy
spec:
  description: "Alerts on any changes to global network policy"
  summary: "[audit] [privileged access] change detected for ${objectRef.resource} ${objectRef.name}"
  severity: 100
  period: 5m
  lookback: 5m
  dataSet: audit
  query: (verb=create OR verb=update OR verb=delete OR verb=patch) AND "objectRef.resource"=globalnetworkpolicy
  aggregateBy: [objectRef.resource, objectRef.name]
  metric: count
  condition: gt
  threshold: 0

Cloud API attacks

The following configuration detects any attacks on a cloud API from a default namespace. It can be modified for cloud etcd, RBAC, and other cloud resources.

• Detects any requests made to cloud provider’s metadata API from application namespace
• This is very suspicious within a cloud environment if the namespace isn’t supposed to use this API for any legitimate activity

Manifest 6:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: metadata-api
spec:
  nets:
  - 169.254.169.254

---

apiVersion: projectcalico.org/v3
kind: GlobalAlert
metadata:
  name: global.metadata.requests
spec:
  description: "Metadata API: Connection to metadata service from default namespace"
  summary: "[flows] [Metadata access] ${source_namespace}/${source_name_aggr} accessed metadata api"
  severity: 100
  period: 10m
  lookback: 10m
  dataSet: flows
  query: '(dest_name_aggr="metadata-api" OR dest_name_aggr="kse.kubernetes") AND action="allow" AND proto="tcp" AND reporter="src" AND (source_namespace="default")'
  aggregateBy: [source_namespace, source_name_aggr]
  field: num_flows
  metric: sum
  condition: gt
  threshold: 0

The earlier attacker made a request to the cloud metadata service from compromised Drupal-pod to determine token privileges and Calico Enterprise Global Alerts detected the attempts as follows:

Snort IDS engine is a single click deployment to monitor containers for malicious activity. Custom snort signatures can be added as given below through configmap. Once the malicious activity is detected inside container traffic, the alerts with necessary details and Kubernetes context is provided for incident response.

Enable snort engine for a specific namespace by deploying the DeepPacketInspection resource:

apiVersion: projectcalico.org/v3
kind: DeepPacketInspection
metadata:
  name: webmin-dpi
  namespace: crown-space
spec:
  selector: all()

If needed, one can create cusom rulesets for snort engine. Following custom snort rule selects all the ICMP Echo requests and responses from a container.

apiVersion: v1
kind: ConfigMap
metadata:
  name: localrule
  namespace: tigera-intrusion-detection
data:
  rules: |
    alert icmp any any -> any any (msg:"ICMP Echo Request"; itype:8; sid:1000000;)
    alert icmp any any -> any any (msg:"ICMP Echo Reply"; itype:0; sid:1000001;)

Conclusion

With Calico on AWS EC2 self-managed Kubernetes and EKS, you can monitor any Kubernetes resource or situation for Pod/deployment/daemonset/cronjob/job/webhook/networkpolicy (any Kubernetes object), failed requests to Kubernetes API server, failed attempt to create roles/rolebindings/clusterrole and clusterrolebindings, anonymous access to k8s api server and, service account access by public IP. You can detect the attack and produce a high-value alert with information that enables the team to readily analyze the threat and prevent the attack from propagating.

Using Calico and Global Alerts together, users can easily monitor their services and applications in AWS EC2 and EKS, quickly identify malicious behavior, and significantly reduce time to address the security threats. Users can address the following:

Calico with Global Alerts on AWS and EKS provides advanced threat and lateral movement detection capabilities in AWS and EKS for preventing any malicious activities in your environment.

To try it yourself, register for Calico Cloud on the AWS marketplace.