The microservices architecture provides developers and DevOps engineers significant agility that helps them move at the pace of the business. Breaking monolithic applications into smaller components accelerates development, streamlines scaling, and improves fault isolation. However, it also introduces certain security complexities since microservices frequently engage in inter-service communications, primarily through HTTP-based APIs, thus broadening the application’s attack surface. This scenario is similar to breaking a chunk of ice into smaller pieces, increasing its surface area. It is crucial that enterprises address these security challenges before benefiting from adopting a microservice architecture.
Challenges implementing defense-in-depth for containers with perimeter-based Web Application Firewall
Kubernetes is the de-facto standard for microservices orchestration. However, as organizations increasingly adopt Kubernetes, they run the risk of inadvertently introducing security gaps. This is often the result of attempts to integrate traditional security tooling into a cloud-native ecosystem that is highly dynamic, ephemeral, and non-deterministic. Instead of implementing security around the platform, DevOps, security, and platform teams must look at enforcing defenses through the platform.
Let’s look at an example of a web application firewall (WAF) which is typically deployed at the ingress of a network or application. As shown in the diagram below, HTTP traffic is inspected as it traverses the WAF before reaching the service in the Kubernetes cluster, which then forwards the traffic to the backing pod (workload). This architecture worked well as a perimeter solution with traditional infrastructure when the network path for a packet was deterministic, and networking teams could ensure that traffic is always routed via the WAF for a given destination as it originates from an external source. The challenge lies in maintaining this level of control in a more dynamic environment characteristic of modern, cloud-native infrastructure.
Let’s now look at how an attacker could bypass such perimeter controls in a Kubernetes cluster. The most common Kubernetes attack vectors can be categorized into three main types,
- External attackers
- Malicious containers
- Compromised or malicious users
Reference: https://www.cncf.io/blog/2021/11/08/kubernetes-main-attack-vectors-tree-an-explainer-guide/
Attackers initially attempt to establish a foothold in the environment by exploiting compromised applications or malicious containers. A Kubernetes cluster consists not only of applications developed by internal teams but also various platform-level components and third-party tools deployed as containers, which could be susceptible to the growing threat of software supply chain attacks. These tools often establish connections with external services or make call-home requests, further increasing the attack surface. Perimeter defenses become ineffective once an attacker successfully infiltrates the cluster. As shown in the diagram below, an attacker can laterally target any service in the cluster and gain access to sensitive workloads while staying under the radar and evading detection. Furthermore, once inside the cluster, attacks could also target internal APIs, for which security tends to be more lax than APIs exposed externally. To adequately protect their environments, security teams must adopt an “assume breach” mindset when designing defenses, recognizing that perimeter-based solutions alone are insufficient in such a threat landscape.
Workload-centric Web Application Firewall with Calico
Kubernetes network security policies can limit the blast radius of a breach by enforcing least-privilege access to services in the cluster. Calico offers a robust security policy model for identity-based microsegmentation that provides the first line of defense to stop attacks in their tracks. For this example, however, let’s assume that the attacker was able to impersonate a legitimate user or workload and initiate an attack circumventing the security policy. The next line of defense involves inspecting HTTP packets for attack detection and mitigation. Calico takes a Kubernetes and container-native approach to application layer security that helps security teams fully embrace defense-in-depth principles for Kubernetes.
Once Calico’s workload-centric WAF is activated for a specific service, traffic for that service is routed via a Calico Envoy-based proxy. This proxy performs attack detection and mitigation based on the content of the HTTP header and body. Calico uses a sidecar-less approach that offers unparalleled flexibility, allowing users to transparently enable application layer security, with as much effort as a click of a button, for any HTTP or gRPC-based service in the cluster.
Deploying Application-level Security
Calico implements a dual approach to application-level security.
- Application Layer Policy (ALP) can look at HTTP methods and path prefixes to restrict invalid requests based on HTTP header attributes.
- Web Application Firewall (WAF), based on OWASP ModSecurity Core Rule Set, can inspect HTTP header and body for attack detection and mitigation.
Application-level security in Calico can be deployed on-demand for the required services. DevOps or security teams can enable the feature by deploying the ApplicationLayer custom resource.
kind: ApplicationLayer
metadata:
labels:
name: tigera-secure
spec:
applicationLayerPolicy: Enabled
logCollection:
collectLogs: Enabled
logIntervalSeconds: 5
logRequestsPerInterval: -1
webApplicationFirewall: Enabled
Users can either add an annotation to a service definition or use the Calico UI to enable application level security on a per service basis. The screenshot below demonstrates how this can be enabled using the Calico UI.
Application Layer Policy (ALP)
An Application layer policy (ALP) restricts traffic based on HTTP methods and paths in the HTTP header. Before implementing an ALP, let’s look at some common attack techniques involving the HTTP header.
HTTP method tampering
According to Common Attack Pattern Enumeration and Classification (CAPEC) 274 in an HTTP method or verb tampering attack,
“an attacker modifies the HTTP Verb (e.g. GET, PUT, TRACE, etc.) in order to bypass access restrictions. Some web environments allow administrators to restrict access based on the HTTP Verb used with requests. However, attackers can often provide a different HTTP Verb, or even provide a random string as a verb in order to bypass these protections. This allows the attacker to access data that should otherwise be protected.”
Reference – https://capec.mitre.org/data/definitions/274.html
The image below shows an example of HTTP methods used by the user agent for an application. We see that the GET and POST methods are typically used.
Attackers may perform HTTP method enumeration to identify the methods supported by an application. The presence of additional methods can expose vulnerabilities, such as weak access control, that could be exploitable. The image below shows an example of HTTP method enumeration where OPTIONS and HEAD methods were detected in addition to the GET and POST methods. The HEAD or even a made-up method could be used to bypass authentication or authorization if an application redirects users to a login page with a 302 code when the user tries to access it directly. The detection of such atypical methods for an application can be used as indicators of compromise (IoC). It is essential that HTTP requests are logged and monitored to identify such indicators.
HTTP Directory Browsing
Attackers may use automated tools to browse the location of sensitive data in a web application. In a forced browsing attempt, an attacker aims to enumerate and access resources not referenced by the application but are still accessible. Scanning tools like Nikto can search for existing files and directories based on well-known directories and resources. CWE-425 states that web applications susceptible to such attacks often make the false assumption that such resources can only be reached through a given navigation path and only apply authorization at certain points in the path ( Reference – https://cwe.mitre.org/data/definitions/425.html).
Implementing Application Layer Policies (ALP)
Calico Application Layer Policy (ALP) inspects HTTP header data to allow or deny incoming packets based on the method and the path in the request URL. Next, let’s look at a couple of Application Layer Policies.
Example 01 – Restrict HTTP Methods
Let’s consider an example application called bookinfo with a frontend service called productpage. An HTTP request for the OPTIONS methods indicates that the HEAD, OPTIONS, and GET methods are allowed. The outputs are shown below,
curl -v -X OPTIONS productpage.bookinfo.svc.cluster.local:9080
curl -v -X OPTIONS productpage.bookinfo.svc.cluster.local:9080 * Trying 10.0.193.61:9080... * Connected to productpage.bookinfo.svc.cluster.local (10.0.193.61) port 9080 (#0) > OPTIONS / HTTP/1.1 > Host: productpage.bookinfo.svc.cluster.local:9080 > User-Agent: curl/7.88.1 > Accept: */* > < HTTP/1.1 200 OK < content-type: text/html; charset=utf-8 < allow: HEAD, OPTIONS, GET < content-length: 0 < server: envoy < date: Tue, 04 Jul 2023 18:03:02 GMT < x-envoy-upstream-service-time: 1 < * Connection #0 to host productpage.bookinfo.svc.cluster.local left intact
Let’s now implement a Calico security policy only to allow the GET method for requests to productpage. An example policy is given below.
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: default.bookinfo
spec:
tier: default
order: 1
selector: 'app in { "productpage"}'
namespaceSelector: ''
namespaceSelectorIsTextbox: false
serviceAccountSelector: ''
ingress:
- action: Allow
protocol: TCP
source: {}
destination: {}
http:
methods:
- GET
paths: []
types:
- Ingress
- Egress
The user now receives a 403 Forbidden error when the OPTIONS method is used.
curl -v -X OPTIONS productpage.bookinfo.svc.cluster.local:9080
curl -v -X OPTIONS productpage.bookinfo.svc.cluster.local:9080 * Trying 10.0.193.61:9080... * Connected to productpage.bookinfo.svc.cluster.local (10.0.193.61) port 9080 > OPTIONS / HTTP/1.1 > Host: productpage.bookinfo.svc.cluster.local:9080 > User-Agent: curl/7.88.1 > Accept: */* > < HTTP/1.1 403 Forbidden < date: Thu, 29 Jun 2023 13:24:40 GMT < server: envoy < content-length: 0 < x-envoy-upstream-service-time: 3
The example demonstrates how Calico policies can be used to limit the accepted methods even if the methods happen to be allowed by the HTTP server.
Example 02 – Restrict HTTP URLs
For this example, let’s use a vulnerable web application and run an NMAP scan to retrieve a list of sensitive files. A scan reveals the following files,
┌──(root㉿kali-deployment-587c555697-jwt95)-[/] └─# nmap --script http-enum www Starting Nmap 7.94 ( https://nmap.org ) at 2023-07-04 19:11 UTC Nmap scan report for www (10.0.142.139) Host is up (0.000080s latency). rDNS record for 10.0.142.139: www.default.svc.cluster.local Not shown: 998 filtered tcp ports (no-response) PORT STATE SERVICE 80/tcp open http | http-enum: | /login.php: Possible admin folder | /robots.txt: Robots file | /phpinfo.php: Possible information file | /.git/HEAD: Git folder | /README: Interesting, a readme. | /classes/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)' | /documentation/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)' | /images/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)' | /includes/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)' | /javascript/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)' | /labs/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)' | /passwords/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)' |_ /styles/: Potentially interesting directory w/ listing on 'apache/2.4.57 (debian)'
Let’s now implement a security policy that only permits access to the robots.txt file. An example policy is given below.
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: default.www
spec:
tier: default
order: 1
selector: app == "www"
namespaceSelector: ''
serviceAccountSelector: ''
ingress:
- action: Allow
protocol: TCP
source: {}
destination: {}
http:
methods: []
paths:
- exact: /
- exact: /robots.txt
types:
- Ingress
The results of the scan after implementing the ALP can be seen below. The HTTP enumeration only lists the robots.txt file when the ALP is enforced since the path was permitted by the policy.
The example demonstrates how Calico policies can be used to restrict access only to legitimate paths for the service.
Analyzing security threats with application layer Logs
Calico’s application layer logs can be used to identify file access attempts and the responses returned by the server. By analyzing these logs, security professionals can gain insights into the activities and interactions happening within the application layer. The example dashboard below includes a pie chart showing the 403 responses for the scan performed above and the file access attempts made by the NMAP enumeration script. This information can be useful in identifying potential unauthorized access attempts or access violations. Security logging and monitoring failures are among the OWASP top ten gaps that can lead to undetected breaches. Adequate logging is essential for visibility, incident alerting, accounting, and forensics. DevOps and security teams should implement effective monitoring and alerting so that suspicious activities are detected and responded to quickly, minimizing the impact of potential security breaches. Such proactive monitoring and timely response are vital to a robust security posture.
Reference – https://owasp.org/Top10/A09_2021-Security_Logging_and_Monitoring_Failures/
Leveraging Calico’s Workload-centric Web Application Firewall (WAF)
Calico offers Web application firewalling based on the OWASP ModSecurity Core Rule Set (CRS). According to OWASP, CRS aims to protect web applications from a wide range of attacks, including the OWASP top ten, with a minimum of false alerts. The CRS provides protection against many common attack categories, including:
- SQL Injection (SQLi)
- Cross-Site Scripting (XSS)
- Local File Inclusion (LFI)
- Remote File Inclusion (RFI)
- PHP Code Injection
- Java Code Injection
- HTTPoxy
- Shellshock
- Unix/Windows Shell Injection
- Session Fixation
- Scripting/Scanner/Bot Detection
- Metadata/Error Leakages
Next, let’s look at how organizations can implement and tune WAF rules using Calico.
Implementing WAF rules
Users can modify the default CRS rules to cater to specific alerting and security requirements. For this example, we will consider Mutillidae II, a free, open-source, deliberately vulnerable web application (https://owasp.org/www-project-mutillidae-ii/). The example below shows that an attacker can bypass the login page by passing the string `’ or 5=5– ‘ in the username field.
The attack triggered rule 942100, which detects SQLi (SQL injection) attacks via libinjection. A sample log is shown below.
Host:'10.224.0.37' File:'/etc/modsecurity-ruleset/REQUEST-942-APPLICATION-ATTACK-SQLI.conf' Line:'46' ID:'942100' Data:'Matched Data: s&1c found within ARGS:username: ' or 5=5-- ' Severity:'2' Version:'OWASP_CRS/3.3.4'
Next, let’s modify the rule so that such an attempt would be denied.
SecRule
REQUEST_COOKIES|!REQUEST_COOKIES:/__utm/|REQUEST_COOKIES_NAMES|REQUEST_HEADERS:User-Agent|REQUEST_HEADERS:Referer|ARGS_NAMES|ARGS|XML:/*
"@detectSQLi" \
"id:942100,\
phase:2,\
deny,\
capture,\
t:none,t:utf8toUnicode,t:urlDecodeUni,t:removeNulls,\
msg:'SQL Injection Attack Detected via libinjection',\
logdata:'Matched Data: %{TX.0} found within %{MATCHED_VAR_NAME}: %{MATCHED_VAR}',\
tag:'application-multi',\
tag:'language-multi',\
tag:'platform-multi',\
tag:'attack-sqli',\
tag:'paranoia-level/1',\
tag:'OWASP_CRS',\
tag:'capec/1000/152/248/66',\
tag:'PCI/6.5.2',\
ver:'OWASP_CRS/3.3.4',\
severity:'CRITICAL',\
multiMatch,\
setvar:'tx.anomaly_score_pl1=+%{tx.critical_anomaly_score}',\
setvar:'tx.sql_injection_score=+%{tx.critical_anomaly_score}'"
The output shows how the WAF denied a subsequent login attempt using SQL injection.
In addition to alerting and blocking common application layer attack vectors, a WAF also serves as an additional layer of defense by providing virtual patching capabilities that can be used to block exploits to new vulnerabilities as they are discovered in the underlying API. By leveraging virtual patching, the WAF acts as a temporary protective barrier, ensuring that enterprises can maintain their security posture and mitigate the risks posed by emerging vulnerabilities. It serves as an essential stopgap measure until the vulnerabilities are adequately addressed and official patches are implemented. This approach is particularly crucial in situations where patching the underlying software or system may take time due to various factors, such as development cycles, coordination with third-party vendors, or complex deployment processes.
Overall, the combination of real-time attack detection, effective blocking of application layer attacks, and the ability to apply virtual patches make a WAF an invaluable security solution for enterprises, allowing them to maintain their security posture and protect against evolving threats.
Workload-centric WAF logs, dashboards, and optimization
Each web application has its own unique characteristics, functionalities, and potential vulnerabilities. Tuning the WAF is crucial to ensure that the rules cater to the specific needs of the application and its threat landscape. Tuning helps strike the right balance between security and usability, ensuring accurate threat detection while minimizing false positives. Calico’s WAF implementation can seamlessly integrate with CI/CD pipelines so that WAF rules can be tested in lower environments before they are deployed in production environments.
Calico offers extensive application level and WAF logs that can help DevOps and security teams to identify and monitor the behavior of an application. A comprehensive overview of the WAF’s activity helps teams make rule adjustments based on actual data that can be easily digested with the help of dashboards. Such dashboards can also be used for collaboration and communication between security and development teams, increasing alignment between different stakeholders in the organizations.
For example, in the WAF logs shown below, there are alerts indicating the use of HTTP/1.1 protocol in an application which can be brought to the attention of the application owners so that it can be remediated while discounting such alerts in a production environment.
Below is an example dashboard developed using Calico’s workload-centric WAF logs that can be used to detect rule triggers and optimize WAF rules as needed. Teams can gain immediate visibility into different aspects of the WAF’s operations using such a dashboard; for example, they can easily identify passthrough traffic, monitored services, blocked attacks, and the types of attacks. The dashboard leverages WAF logs to generate meaningful analytics, such as attack patterns, traffic trends, and attack frequency, which aids in identifying emerging threats and attack vectors targeting critical workloads.
Moreover, dashboards play a crucial role in incident response. In the event of an attack, teams can swiftly identify and investigate the attack through meaningful visualization, allowing for a faster and more targeted response, reducing the impact of an attack, and minimizing downtime.
Summary
Enterprises can significantly benefit by leveraging the powerful orchestration capabilities in Kubernetes to enforce security controls throughout the platform. Calico’s Kubernetes and container-native approach with Application Layer Policy (ALP) and workload-centric Web Application Firewall (WAF) offers an effective means of achieving application layer security within Kubernetes environments.
Ready to try Calico for yourself? Get started with a free Calico Cloud trial.
Join our mailing list
Get updates on blog posts, workshops, certification programs, new releases, and more!
