The Tesla Compromise & Network Policy

Heptio’s CTO, Joe Beda, recently posted an insightful blog entry discussing the Tesla Kubernetes compromise. I wanted to dive into one of the areas he mentioned, network policy.

Before I do, however, I would make some general observations.  While, in hindsight, the vulnerabilities that existed (assuming our analysis is correct) seem obvious, very often in the journey to Kubernetes the security hygiene is left for “when we go production.”  However, very often that journey to production isn’t a well-defined event. An organization progresses from “playing in the lab” to “PoC” to “Beta” to “First Application” and only in hindsight do you realize that you are already “In Production.”  At that point, introducing security mechanisms can be more problematic, maybe even disruptive, and can cause delays in deployment.

I have no idea if that is the case here, but it is something that I see from time to time in the Kubernetes journey.

Security is not just for production

A good maxim is to PoC what you plan to deploy.  You can play fast and loose by not deploying role-based access controls (RBAC) or having lots of components with elevated privileges in production, but you risk some nasty consequences.  It’s best to put some basic hygiene in place before you go into PoC. This also ties in nicely to Joe’s point that security is not just for production.

Now, let’s move on to network policy, which Joe highlighted as a key component in securing your environment and the the workloads hosted in it.  Open source Project Calico, managed by Tigera, is one of the most commonly deployed network policy platforms in use in the Kubernetes ecosystem.  One of the key concepts underpinning Calico and Tigera’s commercial offering – Tigera Secure, is that not only should you protect the workloads from the rest of the world, you should also protect the rest of the world from your workloads.  With rapid development and proliferation of open source packages as the foundation for most modern code, you need to make an assumption that your environment and the workloads within it may not be completely trustworthy. This is the concept of “Zero Trust.”  If you need to have a level of trust in a component, you must build that trust. It also means that you should only apply as much “Trust” as you need to a given component or workload in order to get the job done.

In line with that approach, Project Calico, and by extension, Tigera Secure, enable you to apply a “least trust” model. This model enables you to specifically identify the set of communication flows that are allowed in the platform using an intent-based model tied to the metadata labels already utilized by Kubernetes for the operation of the system.

For example, Joe points out in his post that you can use Network Policy to control which workloads in your cluster can access the dashboard.  As he states, that allows you to limit the workloads that have access to the dashboard to only those that you trust, reducing the risk of a successful exploit if there is a bug or unintentional configuration error in the dashboard that grants the dashboard an unnecessary escalated privilege.

There are some goodies that Network Policy enables:

• Authenticating Proxies: Joe made a very good suggestion to use an authenticating proxy, such as the oauth2-proxy to provide an authentication step before access is allowed to the dashboard.  Calico network policy can strengthen this either in front of and/or behind the proxy.  In front of the proxy, Calico network policy, using either host endpoints (dedicated proxy host) or standard Workload Endpoints (the proxy is a pod), can limit the address ranges that are allowed to connect to the proxy (say to your office, or other trusted networks).  Behind the proxy, you can insure that the proxy can only connect to the desired resources, such as the dashboard. A multi-part discussion of host endpoint policy enforcement can be found in this set of blog posts.
• Controlling Access: Calico network policy in front of the dashboard service can insure that the only thing that can connect to the dashboard is that authenticating proxy.
• Network Policy for Services: If you instead decide to expose the dashboard via the NodePort service model, Calico network policy in host endpoint mode can limit what other resources and/or IP addresses are allowed to connect to that node port service.  We’ve discussed this capability in an earlier blog post.

Joe outlines good steps to secure not only the dashboard, but also many other parts of your Kubernetes infrastructure.  As he highlights, network policy can and should be a part of that prescription. Some of the unique capabilities of Calico’s implementation of Kubernetes network policy provide even more options to improve the security posture of all of your clusters, and in a kubernetes-friendly manner, secures your clusters while facilitating Kubernetes operations.