Why Does It Take Four Months to Get a Firewall Rule Change?

This is the mystery of modern application development and deployment. You have a beautiful new application that is ready to go, but it requires ACL rule changes. Why does it take weeks (or often, months) to get a simple ACL change made in your infrastructure?

The Bad

You’ve invested a lot to make your application development and deployment environment faster and more responsive.  You can deploy an application in under 5 seconds. But let’s face it, it takes you a month to get the firewall change request processed. You probably ask yourself, “what was the point?”  One of our customers recently told me that before Tigera, it took over 4 months to get a firewall rule changed, (a record as far as I’ve heard) but “if you know someone and pull some strings, you can get it done in less than a month.”  What’s going on here?  Believe it or not, the network security team isn’t ignoring you, they just have an extremely difficult problem to manage.

The Ugly

In most firewall platforms, the rulesets are not specific to any particular workload, they are really just lists of IP source and destination addresses, and what ports are allowed between them.  There is no indication of what those IP addresses were when the rule was written, let alone what they are now, 5 years later.  There is also no real grouping as to the function of the rule.

Consequently, every proposed rule change must be vetted against a subset of the existing rules and policies, or in pathological cases all other rules, to ensure that the change does not impact any other policies that are being expressed in the firewall.

Sometimes the firewall team does this themselves.  Sometimes there are multiple parties involved, and the proposed change has to be passed in a waterfall fashion through all of the interested groups for their approval.  If any single group disapproves, or wants a modification, the whole process has to begin again.  In almost all cases this is a painfully manual process.

The Good

There is hope! The cloud native revolution has brought with it some concepts that might make this a tractable, automatable problem.

Orchestration systems such as Kubernetes encourage the deployment of decoupled microservices where each workload on the network is smaller and simpler, with fewer connections than a monolithic app. Each microservice can be labeled with its function, origin or provenance,and so forth.With these labels, systems like Tigera Secure, and Tigera’s open source solution, Project Calico, can build security rules that are inherently bound to the functions of the workloads they are meant to protect; and are easy to understand, evaluate, and modify; now and years from now.  This solves questions that the network security team routinely have to answer such as “What is this rule supposed to do?” and “What does this rule REALLY do?”

While this goes a long way to solving the firewall waterfall model, it doesn’t completely address it.  Without some form of hierarchy and grouping, each organization or group would still need to either evaluate every change to insure that there are no conflicts or be assured by some other party that there are no conflicts.  The evaluation would be easier (and therefore, faster), but it would still be required.

With Tigera Secure, Tigera has introduced a concept called Policy Tiers.  The description of what it is, and how you can use it can be found here, but the short story is that it allows different user groups (Dev, DevOps, Security) to have different, prioritized, network policy groups.  Policies at a higher priority will override the policies at a lower priority.  So long as some common-sense guidelines are followed (the use of pass instead of allow rules in higher priority policies, for example), the use of Policy Tiers can reduce or even eliminate the need for the waterfall.

As an example, let’s say there are just two tiers in the organization, the security compliance team and the developer groups.  The security compliance team is responsible for enforcing HIPAA, PCI DSS, and export controls and similar regulatory requirements, and the developers are responsible for protecting their individual applications.  A diagram describing this behavior can be seen below.

By placing the security compliance team in a higher-priority tier compared to the developers, both groups can edit and change their policies without interfering with each-other.  The compliance team knows that only traffic that meets the current compliance requirements will be “passed” to the developer tier for final determination about allowing or denying the packet.  If it doesn’t pass the compliance rules, it won’t ever make it to the developer tier.

While this use case (firewall rule change latency) is not the one we usually talk about when talking about Policy Tiers, it is a very valuable one.

Tigera Secure, removing one waterfall process at a time, helps provide more secure and  more agile application delivery!