Researchers at Netflix and Google recently reported a vulnerability in the HTTP/2 protocol that enables adversaries to execute a DOS attack by legitimate use of the protocol. These types of attacks are very difficult to detect and mitigate because the traffic is valid HTTP/2 traffic. While HTTP/2 is a relatively new protocol it should be noted that even after several years of hardening we still see vulnerabilities for the TCP protocol like the recently reported SACK vulnerability.
Vulnerability Scanning and Patching
So how do we ensure that Kubernetes workloads are protected from these types of vulnerabilities? Security researchers work to identify new vulnerabilities and then help developers develop security patches. You can apply those patches to keep your software secure from the latest known vulnerabilities.
The simple answer then is to scan workload images and patch your software and update your software to use the latest patches. However, that approach essentially means you have to wait for the next attack and then will need to repeat the cycle. While this works, it is not sufficient and quite disruptive to implement as we play into the hands of the adversaries where they are working on the next vulnerability while users are busy patching the previous one. Also, vulnerability scanners will report vulnerabilities in the main software package and also in associated dependent software packages. In this scenario, the user may not have direct control on the patch and will need to wait for updates from third-party vendors to be available and then plan for a patch upgrade. Due to this, users tend to focus on each vulnerability and ask the question “how does this impact my cluster?” and then try to figure out the plan to patch the vulnerability. This is not an easy question to answer, also the fact that various container scanning solutions don’t always agree on the severity and the CVE score (operational impact of the vulnerability) makes it even more difficult.
Perimeter Firewall
So how do you find an effective method to protect against exploits for these types of vulnerabilities? In the traditional approach, we start to look at leveraging DPI at the ingress to understand the HTTP/2 protocol and then add signatures to detect and mitigate malicious use of the HTTP/2 protocol. This can be achieved but it is not easy to do this from a perimeter firewall. If you look at the HTTP/2 vulnerability in golang-1.11, the HTTP/2 implementation requires the use of TLS. This means a perimeter firewall will now need to decrypt all TLS traffic and then run DPI to analyze and detect the malicious use of the HTTP/2 protocol. This will likely introduce latency (assuming it is feasible to do) and degrade your application performance. Also what about the case where an attack originates from inside the cluster (via an exploited workload), in that case, the perimeter firewall will not even see the anomalous traffic.
Layered Defense
I believe the answer is in the declarative nature of Kubernetes. In Kubernetes, the specification includes the outcome you need for a given application workload (e.g. service deployment, replica-set or even a single pod) and the platform takes care of details like the networking of pods (connectivity and reachability), traffic flow, pod lifecycle management, etc. There is a clear opportunity for security platforms that are native to Kubernetes to be able to efficiently control and monitor workloads in a given Kubernetes cluster. Instead of focussing on a particular protocol or a vulnerability, the approach needs to be to mitigate the impact to the network from vulnerabilities. The right way is to leverage network policy to ensure that all workloads in your cluster have the required amount of access and you can easily do that by leveraging a policy that uses selectors (labels and endpoint selectors) and allows you to enforce policy. If your platform is native to Kubernetes, then it is possible to add context to traffic flow logs to detect anomalous flows. A layered approach is very effective in detecting exploits from these types of vulnerabilities. Tigera Secure allows you to implement this approach as described below.
Layer 1: Network Policy
The first line of defense is to reduce the attack surface. This can be achieved by a simple network policy leveraging a selector to select the right pods (based on services, deployments, etc) to allow ingress from trusted sources and egress as needed. This will limit access to services that are using HTTP/2 to only trusted source pods inside the cluster. This seems simple but many users do not think about this and focus only on building the application and getting it operational. A simple example of how to isolate pods based on network policy is here. A simple policy config shown below will ensure this type of isolation.
kubectl create -f - <<EOF
kind: NetworkPolicy
apiVersion: extensions/v1beta1
metadata:
name: access-nginx
namespace: policy-demo
spec:
podSelector:
matchLabels:
run: nginx
ingress:
- from:
- podSelector:
matchLabels:
run: access
EOF
Note: The NetworkPolicy allows traffic from pods with the label run: access to pods with the label run: nginx. These are the labels automatically added to pods started via kubectl run based on the name of the Deployment.
An example of egress is as follows
kubectl create -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-egress-to-advance-policy-ns
namespace: advanced-policy-demo
spec:
podSelector:
matchLabels: {}
policyTypes:
- Egress
egress:
- to:
- podSelector:
matchLabels:
run: nginx
EOF
Layer 2: Detect anomalous flows
After we achieve the isolation mentioned above, we still need to deal with the scenario where the business need is to expose some services to the outside world. Here you can leverage visibility to effectively detect anomalous flows and mitigate the impact of a given exploit. The security platform needs to be able to clearly distinguish external (e.g. Internet HTTP traffic) and HTTP traffic from inside your organization. Once you have this, you can leverage the context-rich flow logs to run anomaly detection jobs on these flows to detect anomalous traffic. This approach is an order of magnitude more efficient and effective as some classification is happening at collection time. Tigera Secure correlates of the original source IP to the flows that are destined for services inside the cluster via the ingress controller. For flows originating inside the cluster, it provides source pod metadata, like namespace, label, pod prefix, etc. This can allow you to distinguish flows that are from the outside to flows that originated from inside the cluster with very simple queries. This will then allow you to run different anomaly detection algorithms to effectively detect DOS attacks. Note that since the application is native to the Kubernetes cluster, flow log data is enriched at collection time with minimal overhead. This allows for simple queries to get the relevant data for any use case, no need for a large data lake and batch jobs just to add context (correlation) to flow logs.
For example, jobs like the following can be effectively run on the relevant flows to detect anomalous traffic. The platform also allows for the creation of a custom job based on a given requirement.
Inbound connection spike
Job ID: inbound_connection_spike
Looks for pods that receive a higher than the usual number of inbound flows. This may indicate a denial of service or other attack. The job looks for pods that are unusual with respect to their replica set, and replica sets which are unusual with respect to the rest of the cluster.
IP Sweep – External
Job ID: ip_sweep_external
Looks for IPs outside the cluster that are sending packets to a large number of destinations within the cluster. This may indicate an attacker gathering reconnaissance on the active IP addresses in your cluster.
Layer 3: Block Suspicious IPs
Network Policy and detection of Anomalous flows reduce the attack surface for exploit of the vulnerability. In order to further mitigate the impact of the exploit, it is strongly recommended that you leverage up to date threat intelligence feeds to detect and protect your cluster from malicious connections. Tigera Secure allows for the consumption of threat feeds used to detect Suspicious IPs. It should be noted that it is fairly standard to leverage suspicious IP feeds to block malicious traffic, however, Tigera Secure converts the output of these raw feeds into kubernetes resources. This seamlessly enables the use of threat feeds in Policies as “Network Sets”. In the case of the HTTP/2 vulnerability-based exploit, once you have segmentation in place and leverage flow logs to detect anomalous flows, you can create custom network sets based on the anomalous flows to block suspicious IPs to mitigate the impact of the exploit. “Network Sets” can be used as selectors in policies and the network set metadata (labels) is captured in flow logs as additional context. For example, if we see any flows with source/dest IPs matching these network sets, they are logged with the network set metadata. This allows a query of suspicious flows using the network set metadata (label or name) instead of a complex match for an IP in a CIDR or an IP list or matching a domain in a domain list. Note that Tigera Secure Enterprise leverages Kubernetes constructs to efficiently mitigate threats. Listed below is an example of how easy it is to create a network set with a threat feed and use it in a policy
apiVersion: projectcalico.org/v3
kind: GlobalThreatFeed
metadata:
name: sample-global-threat-feed
spec:
content: IPSet
pull:
http:
url: http://an.example.threat.feed/blacklist
globalNetworkSet:
labels:
security-action: block
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: default.blockthreats
spec:
tier: default
selector: all()
types:
- Ingress
ingress:
- action: Deny
source:
selector: security-action == 'block'
Summary
A security platform should allow you to easily enable multi-layer protection described above for your workloads, so you can build a secure cluster with minimum operational overhead. The layered approach blocks a lot of avenues for an attacker to exploit the vulnerability since the attack surface is significantly reduced. This way an attack can be mitigated even before the next vulnerability is detected! The Kubernetes platform is flexible and allows for this layered approach with minimal disruption to normal operations. The key is that the platform needs to be native to Kubernetes to make it a very efficient operation for the user. Tigera Secure is a cybersecurity software that is powered by Calico, which is proven to implement networking and network policy at scale for Kubernetes clusters.
- Segment (via network policy) the cluster to allow access (ingress or egress) to workloads as needed.
- Leverage the Kubernetes native platform to get context-rich flow logs and build effective anomaly detection jobs (no need to build a cluster to analyze logs from your production cluster)
- Leverage Threat Intelligence Feeds that are integrated into Network Policy so you can stay current with threat defense
This simple layered approach can help your cluster be prepared to handle the next set of similar vulnerabilities!
————————————————-
Free Online Training
Access Live and On-Demand Kubernetes Tutorials
Calico Enterprise – Free Trial
Solve Common Kubernetes Roadblocks and Advance Your Enterprise Adoption
Join our mailing list
Get updates on blog posts, workshops, certification programs, new releases, and more!
