Detect unknown network threats with Calico Cloud Honeypods

What are honeypods?

Based on the well-known cybersecurity method, “honeypots”, Calico Cloud runtime security approach of Honeypods as decoy pods are designed to attract traffic to them from malicious sources and to detect suspicious activity within a Kubernetes cluster. Once the decoy pods and services masquerading as a sensitive asset are deployed in different namespaces and global alerts are configured, any attempts to communicate with them can be considered indicative of a suspicious connection, and the cluster may be compromised. Platform or DevOps administrator can then take action following any alerts to immediately quarantine the affected pods and take further action to minimize the fallout of a potential attack with minimal critical business impact.

Calico Cloud, with its network and container-based runtime security, has made the cybersecurity method of honeypods be easy to deploy, manage, and operate at workload level.

Once a cluster is connected to Calico Cloud, honeypods can be deployed on it using ‘template’ manifests that can be easily customized for the honeypods and sample global alerts.

The following steps deploy a set of honeypods in a namespace called tigera-internal and set up global alerts for a few example scenarios.

Configure namespace and RBAC

Apply the following manifest to create the namespace and RBAC for the honeypods:

kubectl create -f https://downloads.tigera.io/ee/v3.18.0-1.1/manifests/threatdef/honeypod/common.yaml

Add the Tigera pull secret into the tigera-internal namespace:

kubectl get secret tigera-pull-secret --namespace=calico-system -o yaml | sed 's/namespace: .*/namespace: tigera-internal/' | kubectl apply -f -

Deploy honeypods

Scenario: IP Enumeration

Deploy and expose an empty pod that can be reached only by PodIP so that you can see when an attacker directly probes the pod network:

kubectl apply -f https://downloads.tigera.io/ee/v3.18.0-1.1/manifests/threatdef/honeypod/ip-enum.yaml

Scenario: Expose a web service

Expose a nginx service that serves a generic page. The pod can be discovered via ClusterIP or DNS lookup. An unreachable service tigera-dashboard-internal-service is created to entice the attacker to find and ultimately reach the tigera-dashboard-internal-debug service:

kubectl apply -f https://downloads.tigera.io/ee/v3.18.0-1.1/manifests/threatdef/honeypod/expose-svc.yaml

Scenario: Vulnerable MySQL service

Expose a SQL service that contains an empty database with easy access. The pod can be discovered via ClusterIP or DNS lookup:

kubectl apply -f https://downloads.tigera.io/ee/v3.18.0-1.1/manifests/threatdef/honeypod/vuln-svc.yaml

Verify the deployment

To verify the installation, ensure that honeypods are running within the tigera-internal namespace:

kubectl get pods -n tigera-internal

NAME                                         READY   STATUS    RESTARTS   AGE
tigera-internal-app-fk87k                    1/1     Running   0          27h
tigera-internal-app-txwrp                    1/1     Running   0          27h
tigera-internal-dashboard-6b78786795-vcbz4   1/1     Running   0          27h
tigera-internal-db-55c75ff886-5r8nm          1/1     Running   0          27h

Verify that the global alerts exist as well:

kubectl get globalalerts

NAME                   CREATED AT
honeypod.fake.svc      2023-10-24T17:45:22Z
honeypod.ip.enum       2023-10-24T17:45:21Z
honeypod.network.ssh   2023-10-24T17:45:21Z
honeypod.port.scan     2023-10-24T17:45:22Z
honeypod.vuln.svc      2023-10-24T17:45:23Z

Simulate attacks to the honeypods

A popular troubleshooting container image nicolaka/netshoot can be used in a pod to use common network tools to simulate probing attacks on the honeypods that will trigger the alerts that have been set up.

• Deploy the attacker pod:

kubectl run attacker --image nicolaka/netshoot -it

• To trigger alerts for honeypod.ip.enum and honeypod.port.scan, get the pod IP for one of the honeypods and save to a variable:

INT_POD_NAME=$(kubectl get pods -n tigera-internal -l app=tigera-internal-app -o=jsonpath='{.items[].metadata.name}')
INT_POD_IP=$(kubectl get pods -n tigera-internal $INT_POD_NAME -ojsonpath='{.status.podIP}')

• Exec and ping the IP from the attacker pod

kubectl exec -it attacker -- ping -c3 $INT_POD_IP

• In the Calico Cloud service graph, we see that there are alerts generated in the default and tigera-internal namespaces that host the attacker and the honeypods respectively.

• Clicking on the namespace and going to the alerts tab in the flow table gives more context:

• The alerts provide context on the description, name of the alert that was triggered and
  the source and destination metadata.
• The alerts can also be viewed under Activity > Alerts

• To trigger the honeypod.fake.svc alert, have the attacker pod curl to the internal debug honeypod service

kubectl exec -it attacker -- curl -v -m3 http://tigera-dashboard-internal-debug.tigera-internal:8888

• The alerts can be viewed again in the service graph and in the alerts page:

• To trigger the honeypod.vuln.svc , curl or trigger a port scan using netcat from the attacker pod to the vulnerable MySQL service:

Netcat:

Curl:

curl http://tigera-internal-backend.tigera-internal:3306

• The corresponding alerts can be viewed in Calico Cloud:

• To simulate an SSH client request to the honeypod PodIP and trigger the honeypod.network.ssh and honeypod.ip.enum alerts, attempt to SSH from the attacker pod to the honeypod IP

kubectl exec -it attacker -- ssh admin@$INT_POD_IP

• The alerts can be viewed in Calico Cloud:

• At this point, an admin can leverage the information to decide on further action to be taken with the existing workloads in a cluster.

Conclusion

As part of maintaining a strong security posture for the cloud-native application, honeypods serve as a security capability in discovering potential network-based attacks from unknown sources within the cluster by diverting the traffic away from legitimate services and pods. It provides a way for an administrator to take immediate action to minimize the cost of business-critical workloads in the cluster. Calico Cloud provides an easy framework to deploy honeypods and set up alerting and monitoring that can be utilized as a part of the security toolbox to secure a Kubernetes cluster.

Ready to try Calico node-specific policies? Sign up for a free trial of Calico Cloud