How to Avoid Hijacking the Internet (or anything else) with Your Cluster

Tigera’s solutions are equally applicable in both private and public cloud environments. In fact, we are the underlying network policy platform in the hosted Kubernetes offerings from AWS, Azure, Google, and IBM. In almost all cases in the public cloud environment, we utilize the existing networking capabilities (e.g. the native CNI interfaces) of the public cloud provider. However, in the private cloud, there really isn’t a container-aware native networking layer. In those cases, Tigera’s solution not only provides the network policy layer, but basic network connectivity as well.

To enable that network connectivity, Tigera Secure and Project Calico platforms utilize BGP as a routing protocol. This allows the platform to exchange routing information with the existing data center network, obviating the need for an overlay and enabling a hyper-scale environment.

Most network engineers recognize the benefits of this approach, but some become concerned that the Tigera platform could, either through a bug in the code, misconfiguration, or malicious intent of an operator of the platform, inject bogus or false routes into the data center network through the BGP protocol.

The attack/error that I just described is often referred to as “route hijacking.”

We address this risk by keeping security in mind when configuring BGP within the Tigera platforms. One of the key mechanisms we use is BGP route policies. The policies that we configure prevent any Tigera node from announcing any routes that are not covered by one of the configured address pools in the cluster. This prevents any Tigera node from ‘hijacking’ any route in the network.

However, this doesn’t protect against someone who introduces a node not under the control of Tigera into the cluster, or someone who can subvert a Tigera node. In that case, the route policies will not be in place, and a route can be advertised, causing a hijack.

To address this concern, route policies can be configured on the ToR and/or spine switches, and even on the border router, if there are concerns about the cluster switches being subverted. The diagram below shows a network topology, with the ToR, spine, and border routers identified, showing where those filters could be installed.

In this example, the Tigera cluster is colored in orange, and the external, data center interconnection router is in blue.

In Tigera’s solution, one or more IP address ranges are delegated to the cluster, which manages the assignment and routing of those addresses within the cluster. In the case of Kubernetes, for example, the address ranges, or pools, would include the service or cluster range, and one or more pod ranges. This means that the cluster has the exclusive use of those addresses, and is authoritative for those addresses. It also means that it should not try and ‘announce’ or tell the rest of the infrastructure about any other routes than those that involve those ranges.

This allows us to write some BGP route policies to protect against the hijack attack or accident. Let’s assume that there are two pools assigned to the cluster, 191.51.100/24 and 192.0.2.0/24. To prevent the cluster from announcing any other routes than those two ranges, we will develop a few policies. Either one of these would be sufficient, the two, in combination protect against errors in the policy creation, and are really just a set of belt and braces.

In a router you must define the BGP peers you are going to exchange routes with. In this case we are going to talk about the peering between spine switch A and the border router.

On the spine switches in the cluster, we would write an export policy. I’m going to write this in pseudo code, as each router vendor has their own language, but the concepts are the same.

bgp config{
  peer (border_router){
    peer ip (<border router IP>)
    peer as (<border router AS>)
    import (!192.0.2.0/24, !191.51.100.0/24)
    export (192.0.2.0/24, 191.51.100.0/24)
  }
}

The route policies here will allow the spine switch to only send routes from the two pools assigned to the cluster, when exchanging routes with the border router, it also will allow the border router to send any routes other than the cluster pools to the cluster (preventing the reverse hijack).

Reversing the ingress and egress rules on the matching configuration on the border router will block the reception of any route not assigned to the cluster from the spine switch, and allow any routes to be sent to the cluster other than the addresses that the cluster is authoritative for.

With this simple precaution, the rest of the network, and the cluster itself, are protected from route hijacking, either malicious, or by accident.

So, what is a BGP policy anyway?

Routers exchange two different types of traffic or messages.  For example, if router A receives traffic from one of its subtended hosts, say 192.0.2.15 that is destined for 191.51.100.37, it will look in its routing table and, may discover that 191.51.100.0/24 is reachable through router B.  If you want to  prevent that traffic from being sent, a traffic or ‘firewall’ filter could be put in place on either Router A or Router B to deny it.

But, how did Router A know to send the traffic to Router B.  Routers exchange routing information, telling each other what destinations they know how to reach.  A routing protocol is used to do this, and one of these is called BGP.

BGP enables a feature called route policies.  In this case, BGP routers that are communicating across a trust boundary (called an AS boundary) can limit what routes they communicate.  In fact, by default, BGP routers don’t allow any routes to be exchanged unless they are specifically configured.  This is called route policy.  In the case above, unless there is a route policy that allows Router B to tell Router A that it knows how to get to 191.51.100.0/24, Router A will never know that it could send the traffic to Router B in the first place.

————————————————-

Free Online Training
Access Live and On-Demand Kubernetes Tutorials

Calico Enterprise – Free Trial
Solve Common Kubernetes Roadblocks and Advance Your Enterprise Adoption