What is platform engineering and when should you invest in it?

As application platforms grow larger, the idea of DevOps teams where developers support the software development lifecycle, but also manage infrastructure and the platform, is beginning to reach the limits of what these teams can support. Rather than taking their best application developers and making them work on infrastructure problems, more organizations are coming to the conclusion that a centralized platform team specialized in that area is a better use of their developers’ skill sets. But what exactly is the platform engineering team and how is it different from the DevOps team? Should your organization invest in platform engineering? Let’s take a closer look.

Platform engineering: What is it and how did it come about?

Platform engineering is essentially building (selecting/standardizing on), operating, and managing the infrastructure that supports 1st- and 3rd-party applications. In the days before cloud-native application development, what we saw was that there was a central team that provided compute infrastructure for enterprise developers to build and host their applications. At a certain point in time, those developers moved to a microservices-based architecture. They didn’t just need virtual machines or servers where they could run their applications; they were building those applications in a containerized form factor, using Docker for their development infrastructure, and then using some form of orchestrator to run their applications. What they needed was a development platform for containerized application hosting.

By that time, Kubernetes had become the dominant container platform for orchestration. There were others, like D2iq and Docker, but Kubernetes ultimately won the race. The central teams started providing a development platform that included the Kubernetes orchestrator and everything else needed by enterprise developers to build and host their applications.

Today, I’ve seen organizations include things like the repo structure (registries) and any kind of application operations infrastructure (build infrastructure, monitoring infrastructure, data stores, logging infrastructure, etc.) in that platform. I’ve also seen organizations use application-layer infrastructure as part of their platform, including CI/CD pipelines and service mesh. This sometimes includes capabilities around blue-green deployments and canary deployments included as part of the platform. But at its core, the platform is compute infrastructure that’s aligned with a microservices architecture in a true multi-tenant fashion. That’s what platform teams are responsible for managing.

Platform engineering vs DevOps

In many cases, platform engineering is an overlay to DevOps. While DevOps teams tend to work more closely with application teams to understand their operational requirements, platform engineering’s charter becomes standardizing on solutions and infrastructure to support the entire estate (e.g. choosing a consistent data plane). Overall, platform engineering simplifies operations and reduces overhead to standardize on infrastructure components.

With organizations including so many types of infrastructure as part of their platforms, it makes sense for a centralized team to manage it all. If you have hundreds of developers in your organization, it doesn’t make sense to have all of them doing the same set of things. It would be an extreme waste of resources if every single Dev team had to manage their own logging, registry, and operating infrastructure. More importantly, it is very difficult to implement a corporate-wide regulatory framework for security across the platform if every team is doing their own thing. You need a centralized platform, and a specialized team—the platform engineering team—to manage it.

If we look at DevOps as a setup, you can never expect a true full stack software developer to also own systems and infrastructure—this requires a different skill set, mindset, and approach. It’s not realistic to have software developers who are writing application code also write platform code or platform automation scripts and manage the operations. When I have seen DevOps teams owning everything end-to-end, there has always been a specific person on the team with an operational skill set who was owning the Ops aspect of the DevOps. In all other cases, it just doesn’t work.

When to invest in platform engineering

If your organization is entirely in the cloud, and your Dev teams are running their applications on managed Kubernetes services, then a large part of your compute infrastructure and platform is managed by the service provider. In this case, you wouldn’t necessarily need to centralize that function, but there would be bits and pieces of that platform that you would need to run, operate, and manage. That’s where it would be useful to invest in platform engineering.

For example, let’s look at logging across all EKS and AKS clusters in an organization. If you have teams that are building their own Kubernetes clusters through OSS distros, you would absolutely want to centralize that function. Why? Primarily because managing, building, and operating these different distros is not straightforward; it requires a specific set of skills and there is quite a learning curve. You certainly wouldn’t want every Dev team in your organization to have to overcome that learning curve. Also, without a centralized logging infrastructure and a centralized team to oversee it, you run the risk of inefficiency as multiple teams work toward disparate solutions to the same problem. Lastly, having a platform engineering team in this case would enable application developers to focus on innovation and delivering new features that add value for the business.

An even more important reason to invest in platform engineering is that, if you are building everything yourself, you are responsible for managing security, compliance, and uptime for your infrastructure. There are specific requirements and playbooks for operating OS distros securely; you don’t want to have to do this work N number of times with every single team in your organization. This is when you want to invest in a platform engineering team.

The main takeaway here is that, even if your organization is entirely in the cloud, at some point in time you will likely want to invest in platform engineering. Because when you have hundreds of developers building and running applications on cloud-based, managed Kubernetes services, it doesn’t really matter how small the platform component is. When you take into account that the work needs to be distributed across 10, 20, or 30+ Dev teams (depending on the size of your organization), you realize the inefficiency of it all. It makes more sense to centralize that function and have one team responsible for platform engineering.

To learn more about new cloud native approaches for establishing security and observability for containers and Kubernetes, check out this O’Reilly eBook by Tigera.