Multi-VRF support for Egress Gateways using Calico

This is a follow up discussion of some advanced use case scenarios for Egress Gateways. In a previous blog post, Policy-based routing with Egress Gateways, I explained how to achieve connectivity to multiple destinations using policies based on the destination of the traffic.

One of the use cases described was the ability of connecting to different services based on the destination, so we can use a different source IP that can be included in an allowlist for such services. But what happens if the services we need to access have the same IP address range as destination? Policy based routing rules use the destination CIDR as the matching criteria, so it will not work in this situation.

This may happen for example in situations where those service definitions reside within different organizations, or during merger and acquisitions (M&As). Let’s take a look with an example:

Figure 1: Sample topology with services in different clusters

As depicted in Figure 1, we cannot use Policy-Based routing for Egress Gateways in this scenario, as the destination subnets are the same. We can however leverage Multi-VRF support for Egress Gateways to use a single node to provide connectivity to those using different Virtual Routing and Forwarding instances in the kernel through multiple different routing tables.

Multi-VRF support for Egress-Gateway

Multi-VRF support in Calico is implemented through the creation of a new custom resource managed by Calico called “ExternalNetworks”. First, we must enable this parameter in the main configuration resource for Calico (felixconfiguration):

kubectl patch felixconfiguration default --type='merge' -p \
    '{"spec":{"externalNetworkSupport":"Enabled"}}'

These ExternalNetworks would represent Virtual Routing and Forwarding spaces to handle our traffic. To create them, please follow this documentation, and make sure you read carefully the notes about selecting an index for such resources.

Once those resources are created, we can associate different BGP peering connections to them, as in the example below:

apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
  name: red
spec:
  nodeSelector: "egress == 'red'"
  peerIP: 172.16.1.140
  asNumber: 64512
  sourceAddress: None
  externalNetwork: red

As noted in the manifest, this BGP connection would be associated with the ExternalNetwork “red”, and only the Kubernetes node(s) with the proper label will establish this connectivity. The end result is that all routes learnt through this peering will be installed in the corresponding Routing Table as defined in the ExternalNetwork parameter.

This creates the necessary “plumbing” for our traffic to be routed with each node having multiple independent routing tables to handle that connectivity. But how do we indicate that traffic from a specific egress gateway must be forwarded to this peer, so a route lookup is done in a specific table? That is accomplished by a new stanza in the EgressGateway resource called “ExternalNetworks” (see documentation):

apiVersion: operator.tigera.io/v1
kind: EgressGateway
metadata:
  name: egress-red
  namespace: default
spec:
  logSeverity: "Info"
  replicas: 1
  externalNetworks: ["red"]
  ipPools:
  - cidr: "10.50.0.2/31"
  template:
    metadata:
      labels:
        egress-code: red
    spec:
      nodeSelector:
        egress: red
      terminationGracePeriodSeconds: 0

So, whenever a pod needs to reach an external destination which is not part of the cluster:

• The pod will redirect its traffic to its associated egress gateway if any. This is done as in other use cases by means of an annotation at the pod/deployment, or namespace level.
• Then, this egress gateway will use a specific node(s) which has the required BGP peering. The node(s) will install all learnt routes through a peering associated to an ExternalNetwork into a specific routing table that would have been created when such ExternalNetwork was defined.
• Finally, traffic traversing the Egress Gateway will use this table as configured in its definition to make the needed route lookup for that specific table when forwarding the traffic:

$ ip route show table  500
192.0.2.0/24 via 172.16.1.140 dev ens5 proto bird

Summary

Multi-VRF support for egress gateways provide us the required flexibility to implement security controls in more advanced scenarios, and where a single common routing domain is insufficient.

Ready to get started? Try Calico Cloud for free.