This blog was co-authored by Matt Smith and Christopher Liljenstolpe.
We recently had the opportunity to share the lessons we have learned about container security from deploying Kubernetes and OpenShift in the field. If you don’t have time to watch the full recording of our conversation, here are a few highlights.
It’s important to understand that the processes and people are changing. Do not just stand up a container platform, hand it to people who have existing responsibilities and assume it all just lines up the same.
There are three key elements you will need to coordinate:
Containers aren’t VMs, they’re Linux processes, so you’re moving back to a shared host model. The combination of SELinux, kernel namespaces, capabilities, cgroups, and seccomp provide a strong isolation model – but you have to take full advantage of them. In particular, even if you’ve struggled with SELinux in the past, it’s imperative to use it with containers since the container framework takes the complexity of managing the SELinux details for you.
Further, with Red Hat CoreOS, you also have the option of an immutable, container-optimized operating system, built on the established base of Red Hat Enterprise Linux.
There is a great analogy between containers and sandwiches. You see pre-made sandwiches all the time, and they can be really good and a quick way to grab a bite when you are hungry – but what if you saw a really appetizing sandwich sitting on a park bench, without knowing who put it there or who made it? Would you pick it up and eat it? If you’re getting your sandwich, err … containers, from a third party site, make sure to understand it’s provenance, and determine if it’s trustworthy. That way you won’t accidentally convert your cluster into someone else’s bitcoin mining operation.
Another aspect of container hygiene is that, unlike traditional virtual machine deployment models, you don’t just update them with yum update or apt-get. Instead, the new immutable container deployment model is to rebuild them from scratch, which really needs a fully automated continuous integration process. That CI process should have security built in throughout, including automatic triggering of new builds, vulnerability scanning, and signing of builds so you can be sure of their provenance. When you have hundreds or thousands of containers that all need to be rebuilt due to release of a new critical patch, you need this process to be fully automated.
It’s a standard best practice to not embed passwords or keys within a container image – even if your images are not public, you shouldn’t share such secrets across your organization. And even if you’re just writing a quick-and-dirty proof of concept, you should separate your secrets from the image, to ensure it never makes it to production. Kubernetes includes a secrets management framework, with a pluggable API so you can use an external tool such as Vault, to enable you to do this easily, so there’s no excuse!
With containers being ephemeral, it is no longer possible to hard code rules applying to specific instances. Instead, we leverage labels to create a level of indirection between the policy we want to apply and the workloads currently running in the cluster. Such policies are declarative yaml files, and can be treated just like code – with review and deployment cycles that ensure repeatability and auditability.
There’s no time to sit still. This space is moving quickly and you need to stay on top of the new tools. Things like Istio and Gloo service meshes; or tracing and monitoring tools like Jaeger and Prometheus. In the container space, Red Hat is driving change with CRI-O, buildah, and podman. In the security and compliance space, Tigera is pushing forward the frontier of what is possible in Kubernetes with Tigera Calico and Tigera Secure Enterprise Edition. These tools are all improving the development process and the operational characteristics of the environment, from stability to security to features and functionality. Find some time each week to do a little research, read the latest blogs, and tinker with the technology; otherwise, you and your organization will be missing out on some of the great value these new technologies are providing.
Matt Smith is a Tigera guest blogger. He is the Chief Architect of the Northeast Commercial region for Red Hat, the world’s leading provider of opensource software solutions.
Christopher Liljenstolpe is the CTO Solutions at Tigera.
Get updates on blog posts, new releases and more!