How to detect and stop DDoS attacks in a Kubernetes environment

DDoS or Distributed Denial of Service attacks have been around for a while and are notorious and painful to deal with (as with any other attack). As the name suggests, a DDoS attack causes an application or service to become unavailable to users due to resources exceeding it’s capacity and causing the app to either crash or become unresponsive. DDoS is a form of DoS where the attack comes from multiple sources (bots), usually spread across geographical locations. Imagine getting thousands of spam calls on your phone within a very short time and there is one legitimate call that is trying to contact you. How will you make sure you attend the legitimate call?

In a Kubernetes environment, DDoS can hit the application from external sources when a service is exposed to the Internet. For attackers who gain a foothold of the environment within the Kubernetes cluster and are looking to infect multiple workloads with malware to further amplify a DoS attack, you need a strong zero-trust workload access control policies in place to restrict lateral movement.

Why you should invest in a DDoS solution for your container application

Before we look at the technical aspects of a DDoS attack and how Calico helps with detection and mitigation of such attacks, let us consider the damage it can cause. Although the intent of a DDoS attack is not directly stealing money, data, or installing ransomware, any type of application downtime indirectly translates into monetary loss. Troubleshooting and mitigation efforts also result in lost productivity for teams when they are already burdened with multiple security alerts. Some reasons for deploying a DDoS mitigation solution include:

• Recent uptick of DDoS attacks due to political landscape in war-affected regions.
• Cloud-native applications are designed to scale up automatically when inbound requests spike, resulting in higher usage bills.
• Difficult to protect lateral communication when malware has been successfully implanted for DDoS activities.
• Attackers are becoming more sophisticated with DoS techniques.

How are DDoS attacks carried out?

Threat actors use a variety of techniques as part of a DDoS attack. They can manipulate networking protocols such as TCP to design SYN flood attacks, ICMP (ping flood)-based attack, application layer (HTTP)-based attacks, or DNS amplification attacks. These attacks saturate the wire (network) with a huge amount of packets that causes the routing process to eventually crash and end up in a network downtime. While this blog can provide a good reference about traditional DDoS attack mechanisms and types, there are some newer techniques that threat actors have begun employing by targeting cloud-native applications. Cloud-native applications that are designed to automatically scale resources up and down based on user demand have fallen prey to a new type of DoS attack dubbed as the “yo-yo attack”.

Yo-yo attacks in a Kubernetes-based cloud-native application

An interesting and bothersome attack being carried out on Kubernetes-based applications is the Yo-yo attack. Whenever there is a surge in user activity for an application, Kubernetes automatically scales up the resources (pods, CPU cycles, memory etc) just like an EC2 instance, using either pod autoscaler or cluster autoscaler mechanisms. This has prompted attackers to generate illegitimate requests just so that businesses end up losing money by unwanted scaling of resources up and down (like a yo-yo) without resulting in actual business revenue. This paper unravels the concept in an academic research and shows how this type of DDoS can be classified as an EDoS or Economic Denial of Sustainability attack.

Detecting a DoS attack in Kubernetes using Calico

Calico is embedded into the networking layer in Kubernetes and has access to a rich set of flow logs (Layer 3 and 4), application layer (Layer 7) logs, and DNS logs for all network traffic in the cluster. With this information, we build a baseline behavior for nodes, pods, and services with respect to the amount of traffic that is normal at any given period of time. These logs are further forwarded to Elasticsearch, from which we build behavior-modeling algorithms. Any time there is a deviation from the baseline behavior, an alert is generated informing the user about the deviation.

Calico has a list of anomaly detectors that can predict and detect a denial of service attack. Some of them are high values of http_connection_spike, generic_dns, generic_l7, inbound service bytes anomaly such as bytes_in and few other detectors.

CISA’s recommendations if you are under a DDoS attack and how Calico can help

As a general rule, once you have identified and detected a possible DDoS attack, you could use the recommendations provided by the Cybersecurity and Infrastructure Security Agency (CISA) to perform some of the tasks below from the CISA recommendations to understand the nature of the DDoS attack and how Calico can help with each task.

• What are the ranges of IP addresses used to propagate the attack?
  • With Calico, users get rich Layer 3, 4, HTTP, and DNS flow log information right from the UI dashboard with the Dynamic Service and Threat Graph. This will help identify the IP address information and related context for remediation efforts.
• Look for a particular attack against certain running services.
  • Using Calico’s broad set of container security tools, especially at runtime with anomaly detection, if there is any presence of malicious activity either on the network or the container, our alerting capability can give operators quick and detailed information on any impending threat.
• Correlate server CPU/memory utilization with network traffic logs and application availability.
  • Calico can help users with HTTP flow logs and application-level visibility with various Kibana dashboards and a native Dynamic Service and Threat Graph with visual representation of traffic flows and associated workloads. Combining this with the other metrics such as CPU, memory utilization from the native cloud provider, or any traditional observability tool, can give a complete picture of the attack scenario to start mitigation efforts.
• Once you obtain an understanding of the attack, deploy mitigations.
  • This is where the most important step of deploying Calico security policies come into effect and this is covered in detail in the next section.
• Directly conduct packet captures (PCAPs) of the DDoS activity or work with security/network providers to obtain PCAPs. Analyze PCAPs to verify the firewall is blocking malicious traffic and allowing legitimate traffic to pass
  • Calico’s Dynamic Packet Capture is a true life-saver for teams that want an integrated packet capture tool to perform RCA and troubleshooting when a performance or security issue occurs. All it takes is a couple of clicks and you can customize the capture tailored to your environment and needs.

Responding to DDoS using Calico

Since we are dealing with a DDoS attack in Kubernetes, we will first substantiate if a basic Kubernetes Network Policy can help with responding to a DDoS attack. The default Kubernetes Network Policy does not have the ability to do two things that are critical to stop a DDoS attack in Kubernetes.

• Global Network Policy
• Host EndPoint (HEP) for policy enforcement

These two features are available with Calico, and when combined with Global NetworkSets and XDP offloading, we can effectively stop a DDoS attack before it results in an outage or causes monetary loss.

Before we jump into a real life example of how the policy is created step-by-step, here’s a brief description of the features mentioned above:

Calico Global Network Policy
A type of resource that can be applied to any kind of endpoint (Host End Point in this case) so that a policy for the underlying host can be defined irrespective of namespace.

Host EndPoint (HEP)
Network interfaces on the host where the Kubernetes workloads are deployed. We can use labels with these endpoints and use them within Calico Network policies.

Global NetworkSets
A global resource consisting of a single or range of IP addresses or CIDR ranges or domain names which can send or receive traffic from any network not part of the Calico network. Used in conjunction with Calico Network Policy.

XDP (eXpress Data Path)

XDP is a Linux kernel capability that allows BPF programs to be inserted very early in Linux’s packet processing pipeline. With the ‘XDP Offload’ mode, Calico can provide the fastest blocking performance in case of a DDoS attack.

Step-by-step policy creation using Calico

The first step is to create a Host EndPoint (HEP) with the IP information of the host.

The next step is to create a Global NetworkSet with few IP CIDR ranges as shown.

Finally create the Global Network Policy. Notice how we have used the same label that is defined in the network set previously. Moving forward, this label can be used across multiple policies, even in namespaced Network Policy. We have also set the doNotTrack flag to be true, meaning that this special setting can bypass Linux conntrack and execute the policy rules before the tracking happens so that we don’t drop it.

DDoS mitigation using Calico’s workload-centric web-application firewall (WAF)

Calico leverages Envoy to deploy a WAF using a DaemonAet. This removes a ton of complexity without using a servicemesh approach. The workload-centric WAF support can further strengthen your application’s security posture to protect against DoS attacks. The architectural difference between a traditional WAF and Calico’s workload-centric approach is that any anomaly in the HTTP headers in packets traversing laterally across workloads and not hitting the perimeter of the cluster can be detected and flagged. This enables teams to not only protect the application layer at the edge, but also within the Kubernetes cluster.

The next few steps describe how easily you can enable and deploy a workload-centric WAF for Calico.

Note: Our latest upgrade also supports this configuration right from the Calico Manager UI.

apiVersion: operator.www.tigera.io/v1
kind: ApplicationLayer
metadata:
  name: tigera-secure
spec:
  webApplicationFirewall: Enabled

Conclusion

Since applications in Kubernetes pose an equal, if not greater security risk of  DDoS attacks, organizations require new ways to detect and mitigate these threats. This blog illustrates the various concepts of a DDoS attack within Kubernetes and a few examples of creating a step-by-step mitigation solution using Calico.

If you are interested in seeing Calico in action, attend our free workshops or sign up for a free trial.