Calico Enterprise lets users write network policies using domain names instead of IP addresses. This is done by dynamically mapping domain names to IP addresses and matching the egress traffic against these IPs. We have discussed this feature in detail when we introduced the Inline mode for the eBPF data plane in Calico Enterprise 3.20 release! It addresses the latency and performance issues of the various modes used by Calico in iptables/nftables data planes. It is a shame that Calico users who are not yet ready to switch completely to eBPF would miss out on this big DNS policy improvement. Don’t worry! We found a way to port it to iptables to enhance our users’ experience without forcing users to make a huge leap.
In Calico Enterprise v3.21, we have extended the Inline DNS policy mode to iptables. In this mode, DNS policies are updated in real time as DNS responses are parsed by eBPF within the data plane, thus improving the performance.
Calico iptables – DNS Inline policy
In all the existing modes in the iptables data plane, the DNS response packets are sent to Felix – Calico’s userspace agent. It parses the packets and updates the data plane since advanced packet parsing is not feasible with standard iptables rules. However, iptables has an xt_bpf extension which lets us process and match the packets by an eBPF program the same way we do that in the eBPF data plane!
An iptables rule that allows it may look something like this:
iptables -A INPUT -m bpf --object-pinned /sys/fs/bpf/dns_parser -j ACCEPT
The iptables rule calls the eBPF DNS parser program on the response packet and updates the data plane inline. When the client connects for the first time, the policy is enforced without any delays, thus avoiding any retransmits or any impact on application performance.
The parser program is the exact same as in the case of the eBPF data plane and is called in the exact spot in which the packets were formerly sent to userspace. However, that is only one side of the equation. The conundrum is that the iptables rules that match policy cannot match the eBPF ipset that the parser updates. Vise versa, the parser cannot update the iptables ipset, which the policy rules use.Therefore we replace the iptables ipset matches with eBPF programs that can match against the sets written by the parser.
The outcome
Using eBPF with iptables helps us offer a low latency DNS policy mechanism as part of Calico Enterprise 3.21. This is how Inline policy compares with other DNS policy mechanisms.
Testing workflow
The test creates 3 test pods which will curl to 4 target deployments using
<pod IP>.<namespace>.pod.cluster.local, thereby generating traffic to the DNS policy targets. This is done along with the scale deployment which will be continuously scaled up and down throughout the test to load calico. Further the CPU for calico-node is limited to starve felix and slowdown updates to the data plane. The test is executed for 10mins.
The DNS policy modes are compared against three parameters.
- Duplicate SYNs
- Lookup Time
- Connect Time
DuplicateSYNs estimate the number of retransmits, which indicates how quickly the data plane updates the policy. As we see with Inline mode, there are no retransmits.
Lookup time signifies how quickly the DNS is resolved in the client. As expected, DelayDNSResponse has a lot of delay compared to other modes.
ConnectTime is the time to do a connection to the target once it gets an IP address from the DNS lookup.
Note: Y-axis for the lookupTime and connectTime is logarithmic comparison of values to make it easily understandable.
NFTABLES
Will the same work for nftables? It does not seem like this is going to be the case in the near future. Nftables do not have a native equivalent of the same module. It is possible to use the xtables module—for instance as Cloudflare’s blog describes. Unfortunately, it is being deprecated as well as iptables itself and there does not seem to be an agreement in the nftables and ebpf communities whether adding support for ebpf matchers is worthwhile.
We could use netfilter eBPF hooks to parse the DNS responses, however, the biggest hurdle is that we cannot extend the ipset matches in policy rules as we did with iptables.
Project Calico’s belief always was to use the best tools for the job. eBPF is extremely powerful in extending various parts of the kernel. We think that being able to extend iptables or nftables with eBPF is extremely powerful and combines using a well known system that users understand with custom functionality today rather than waiting for it to make it into the mainline kernels. This is where eBPF really shines!
Conclusion
DNS inline clearly outperforms any previous DNS policy mode and we are happy that we can offer it to the iptables users! This clearly shows that eBPF as a technology can be used along with the legacy, but proven, methods to achieve better results even for users who aren’t ready to fully migrate yet.
However, if you are ready to reap the full benefits of the native eBPF data plane, do not hesitate to follow the Enable the eBPF data plane user guide!
Want to learn more about Calico’s DNS Inline Policy or eBPF data plane? Schedule a demo.
