This blog was co-authored by Matt Smith and Christopher Liljenstolpe.
We recently had the opportunity to share the lessons we have learned about container security from deploying Kubernetes and OpenShift in the field. If you don’t have time to watch the full recording of our conversation, here are a few highlights.
#1 It’s not just technology, make process and people changes too.
It’s important to understand that the processes and people are changing. Do not just stand up a container platform, hand it to people who have existing responsibilities and assume it all just lines up the same.
There are three key elements you will need to coordinate:
- Next Generation Architecture: You’re bringing in new technologies. This is perhaps the most obvious aspect of adopting Kubernetes, but everyone impacted needs to understand what they look like – and I mean everyone. This is not a set of technologies you just hand to one group in your I.T. organization – these technologies are reshaping how the entire organization works.
- New Project Methodologies. Agile is the leader – but there are different models of agile. Make sure you’re adopting processes & methodologies in line with the technology stack, and don’t skimp on the training across the organization!
- Evolving to a more collaborative culture. Embrace transparency. In particular, you have to include the security and compliance teams early in the process. You can’t treat them as a gate to be passed at the end of the project. Everyone in the project, including the developer, needs to think of themselves as a security practitioner, and integrate security right from the beginning.
#2 Containers represent a move back to shared host. Make sure to isolate workloads appropriately.
Containers aren’t VMs, they’re Linux processes, so you’re moving back to a shared host model. The combination of SELinux, kernel namespaces, capabilities, cgroups, and seccomp provide a strong isolation model – but you have to take full advantage of them. In particular, even if you’ve struggled with SELinux in the past, it’s imperative to use it with containers since the container framework takes the complexity of managing the SELinux details for you.
Further, with Red Hat CoreOS, you also have the option of an immutable, container-optimized operating system, built on the established base of Red Hat Enterprise Linux.
#3 Maintain the hygiene of your containers
There is a great analogy between containers and sandwiches. You see pre-made sandwiches all the time, and they can be really good and a quick way to grab a bite when you are hungry – but what if you saw a really appetizing sandwich sitting on a park bench, without knowing who put it there or who made it? Would you pick it up and eat it? If you’re getting your sandwich, err … containers, from a third party site, make sure to understand it’s provenance, and determine if it’s trustworthy. That way you won’t accidentally convert your cluster into someone else’s bitcoin mining operation.
Another aspect of container hygiene is that, unlike traditional virtual machine deployment models, you don’t just update them with yum update or apt-get. Instead, the new immutable container deployment model is to rebuild them from scratch, which really needs a fully automated continuous integration process. That CI process should have security built in throughout, including automatic triggering of new builds, vulnerability scanning, and signing of builds so you can be sure of their provenance. When you have hundreds or thousands of containers that all need to be rebuilt due to release of a new critical patch, you need this process to be fully automated.
#4 Keep secrets out of your container
It’s a standard best practice to not embed passwords or keys within a container image – even if your images are not public, you shouldn’t share such secrets across your organization. And even if you’re just writing a quick-and-dirty proof of concept, you should separate your secrets from the image, to ensure it never makes it to production. Kubernetes includes a secrets management framework, with a pluggable API so you can use an external tool such as Vault, to enable you to do this easily, so there’s no excuse!
#5 This agile methodology isn’t just for applications – * as Code
With containers being ephemeral, it is no longer possible to hard code rules applying to specific instances. Instead, we leverage labels to create a level of indirection between the policy we want to apply and the workloads currently running in the cluster. Such policies are declarative yaml files, and can be treated just like code – with review and deployment cycles that ensure repeatability and auditability.
#6 Investigate the next generation of tools to separate security concerns
There’s no time to sit still. This space is moving quickly and you need to stay on top of the new tools. Things like Istio and Gloo service meshes; or tracing and monitoring tools like Jaeger and Prometheus. In the container space, Red Hat is driving change with CRI-O, buildah, and podman. In the security and compliance space, Tigera is pushing forward the frontier of what is possible in Kubernetes with Tigera Calico and Tigera Secure Enterprise Edition. These tools are all improving the development process and the operational characteristics of the environment, from stability to security to features and functionality. Find some time each week to do a little research, read the latest blogs, and tinker with the technology; otherwise, you and your organization will be missing out on some of the great value these new technologies are providing.
Matt Smith is a Tigera guest blogger. He is the Chief Architect of the Northeast Commercial region for Red Hat, the world’s leading provider of opensource software solutions.
Christopher Liljenstolpe is the CTO Solutions at Tigera.