Another Day, Another Lesson in Securing Your Infrastructure

Last week I wrote a blog post that was a response to an unfortunate event at Tesla.  In it, I detailed some techniques one could use to secure the infrastructure to reduce the risk of suffering a similar security ‘event.’

One of the points I made in that post was that security isn’t “just for production” as production is increasingly a “fuzzy” concept between PoC, PoC with trial customer(s), first application deployment, and so forth.

Right on the heels of that blog post, security researcher Giovanni Collazo published this blog post where he details a fairly simple search for publicly accessible etcd servers on the Internet.  etcd does not ship with user authentication turned on by default.  If you can find an etcd server on the network with a default configuration, you can interrogate it and potentially modify its contents.

Ok, I imagine many of you are wondering “Why this is important, and what is etcd?”  This is where this gets interesting – etcd is a database that was designed to hold configuration data for other servers and services.  The folks at CoreOS (soon to be part of Red Hat) initially wrote etcd to help automate the (re)-configuration of servers and services to ensure that they are kept up-to-date.  This is an admirable, security-driven initiative. It was so admirable that many projects now include etcd as part of their infrastructure, not the least of which is Kubernetes, the very popular container orchestration system that came out of Google and is now the darling of the cloud-native world.

Kubernetes stores lots of “really interesting” data in etcd, as do lots of other projects and users.  In the 2000+ public etcd servers that Giovanni found, he located almost 9000 keys labeled “password” and over 650 keys labeled “aws_secret_access_key.”  I’ll pause here for a minute to allow you to go run and check your etcd servers….

If you install etcd in a default manner, you are basically raining your newly compromised keys on an appreciative, if somewhat unscrupulous, audience.

Let me reiterate the point I made earlier here, and in the previous blog.  Security is not just for production!  If you don’t PoC what you plan to deploy, it is most likely that you will deploy what you PoC’d.

A best practice is to evaluate the components that make up your infrastructure and your application environment and make sure they have rational security settings.  Enabling user authentication is good, enabling mutual TLS (mTLS) authentication for RESTful and similar services (such as etcd) is even better.

With Project Calico’s network policy, and specifically host endpoints, as I discussed in the previous post, you could isolate what endpoints could even connect to your etcd servers.  With Tigera’s CNX, you could go further and control who within your organization can change the policies that control etcd access.

Security starts with some basic fundamental hygiene, like understanding what it is that you are using, how it deploys by default, and what tools are inherent in the component that can improve your security posture.  From there you can build on those basic protections with defense in depth including network policy.

Doing this at the beginning of a deployment is easier, and certainly less disruptive than trying to patch after the fact.  Unfortunately, patching after someone has pulled your “aws_secret_access_key” from your open etcd server is not only disruptive, but also ineffective. The re-keying necessary to recover from that sort of incident (assuming you detect it at all) is more disruptive still.