Today, we announced our collaboration with the Kubernetes networking community on an exciting new project, Istio. In that blog post, we included a video showing how the Istio service mesh could be deployed along with Kubernetes network policies (implemented by Project Calico) to deliver a maximally secure application infrastructure. In this blog, we will look in a bit more depth at how you can start to build such policies yourself, to get the best of both worlds: authenticated, encrypted service-to-service communication running on a fully secured underlying network layer.
Istio
Istio is an open platform to connect, manage, and secure microservices. Istio provides an easy way to create a network of deployed services with load balancing, service-to-service authentication, monitoring, and more, without requiring any changes in service code. The project includes a demonstration application called bookinfo http://istio.io/docs/samples/bookinfo.html to show how a microservices based application may be deployed in a Kubernetes cluster using Istio – and we will use that application as the basis of our example.
Calico
Calico enables networking and network policy in Kubernetes clusters across the cloud. Calico uses a pure IP networking fabric to provide high performance networking, and its battle-tested policy engine enforces high-level, intent-focused network policy. Together, Calico and Kubernetes provide a secure, cloud-native platform that can scale your infrastructure to hundreds of thousands of workloads.
Calico integrates with Kubernetes using CNI and can be used to enforce security policies that are defined in Kubernetes via the Network Policy API.
Securing a Microservices Application
Istio can be used to define and build a mesh of micro services that together compose an application. It can define complex routing (L5-L7) policies between the micro services. For now there is no way to also define the permissible set of intercommunications policies at the layer-3/4 that can be used to secure access to the micro-services application – so to enable Istio to work, you need to turn off network policy enforcement on all nodes. This leaves an undesirably large attack surface to disrupt the application. For example, without network policies in place, it is possible to flood some of the backend micro services-ratings in the case of bookinfo-with spurious requests that can disrupt genuine requests from other micro services.
The rest of this section will show how we can define Kubernetes Network Policies to secure the bookinfo application.
Assuming you have a cluster with the bookinfo application http://istio.io/docs/samples/bookinfo.html already running, you can observe that the productpage, ratings, reviews and details micro services are running.
# kubectl get svc -o wide NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR details 10.108.133.76 <none> 9080/TCP 3d app=details istio-egress 10.99.78.230 <none> 80/TCP 3d istio=egress istio-ingress 10.105.247.10 <pending> 80:32012/TCP,443:30951/TCP 3d istio=ingress istio-manager 10.102.182.17 <none> 8080/TCP,8081/TCP 3d istio=manager istio-mixer 10.109.218.37 <none> 9091/TCP,9094/TCP,42422/TCP 3d istio=mixer kubernetes 10.96.0.1 <none> 443/TCP 5d <none> productpage 10.110.3.182 <none> 9080/TCP 3d app=productpage ratings 10.110.48.206 <none> 9080/TCP 3d app=ratings reviews 10.111.201.116 <none> 9080/TCP 3d app=reviews sleep 10.99.116.199 <none> 80/TCP 3d app=sleep
Setup a default deny policy
The first step in securing any application is defining a default-deny policy for the namespace where the application is running. In this case it will be the namespace ‘default’
# kubectl annotate ns default "net.beta.kubernetes.io/network-policy={\"ingress\":{\"isolation\":\"DefaultDeny\"}}"
Network Policies For Inter-MicroService Communication
Now that we have a default deny policy in place we need to enable communication between the required microservices. The intercommunication path between the services are:
- Istio-ingress to Productpage.
- ProductPage to all versions (v1, v2 and v3) of Reviews service.
- ProductPage to Details service
- And all versions of Review to Ratings service.
Let’s go ahead and specify each individual network policy
Istio-Ingress to Productpage:
#kubectl create -f - << EOF apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata: name: calico-from-ingress-to-ppage namespace: default spec: ingress: - from: - podSelector: matchLabels: istio: ingress podSelector: matchLabels: app: productpage EOF
Productpage to All Versions of Reviews:
#kubectl create -f - << EOF apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata: name: calico-from-productpage-to-reviews namespace: default spec: ingress: - from: - podSelector: matchLabels: app: productpage version: v1 podSelector: matchLabels: app: reviews matchExpressions: - {key: version, operator: In, values: [v1, v2, v3]} EOF
A point to notice here is that we have setup network policy to permit product page to access all versions of the reviews service. Based off how the Istio’s route-rules are setup we can also restrict which specific versions of a reviews service the network policy should be permitted to access.
Productpage to Details service:
#kubectl create -f - << EOF apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata: name: calico-from-productpagev1-to-detailsv1 namespace: default spec: ingress: - from: - podSelector: matchLabels: app: productpage version: v1 podSelector: matchLabels: app: details version: v1 EOF
Reviews to Ratings service:
#kubectl create -f - << EOF apiVersion: extensions/v1beta1 kind: NetworkPolicy metadata: name: calico-from-reviews-to-ratings namespace: default spec: ingress: - from: - podSelector: matchLabels: app: reviews podSelector: matchLabels: app: ratings EOF
Again, here we have defined a network policy that is quite liberal as it permits all versions of the reviews service to access the ratings service.
With all these network policies in place the attack surface to the bookinfo application is significantly reduced.
Further Work
As you can appreciate, the network policies required to secure a micro services application can be quite complex and it is easy to mis-configure the network policies and that may lead to security holes. But the network policies are highly correlated with the route-rules that are defined for intercommunication between the micro services. We at Calico are working closely with the Istio project to automate the generation and insertion of network policies that are derived from the route-rules that define the micro services application.
Also, the policies shown here have the following limitations:
- Only control the ingress path to a service. Using calicoctl commands it is possible to configure network policies that can control the egress paths.
- Do not limit ports that may intercommunicate. Using calicoctl commands it is possible to configure network policies that can be used to restrict inter-communication to specific ports.
- No interactions with external services. In the real world, there are often legacy services outside the cluster that must be included in the application mesh. With Calico it is possible to configure rules to allow access to/from those services, and even deploy an agent on VMs and standalone servers to enforce bookended policies on the legacy application node.
Examples of using calicoctl commands to configure such policies can be found at http://docs.projectcalico.org/v2.2/getting-started/kubernetes/tutorials/advanced-policy
Join our mailing list
Get updates on blog posts, workshops, certification programs, new releases, and more!
