Introduction
Bottlerocket is an open source Linux distribution built to run containers securely in scale by Amazon, it is uniquely tailored to improve stability and performance with a great focus on security. Utilizing a transactional update system Bottlerocket claims to reduce maintenance overhead and lower operational costs. Built from the ground up using the v5.4 Linux Kernel, makes Bottlerocket a great platform to run Calico eBPF.
In this blog post I will go through the necessary steps for setting up an EKS cluster using Bottlerocket OS and Calico eBPF.
About Calico’s eBPF dataplane
Calico’s eBPF dataplane offers performance and capability improvements over the standard Linux networking dataplane, such as higher throughput and lower resource usage, and native Kubernetes Service handling (replacing kube-proxy) with advanced features such as source IP preservation and DSR (Direct Server Return).
If you aren’t already familiar with the concept of eBPF, it allows you to write mini programs that can be attached to various low-level hooks in the Linux kernel, for a wide variety of uses including networking, security, and tracing. You’ll see a lot of non-networking projects leveraging eBPF, but for Calico our focus is obviously on networking, and in particular, pushing the networking capabilities of the latest Linux kernel’s to the limit while maintaining Calico’s reputation for simplicity, reliability and scalability.
If you want to learn more about Calico’s eBPF dataplane, you can find much more information, including performance charts, in this blog.
Before we begin
There are few requirements that you need to have in order to follow this blogpost.
Requirements:
- Install eksctl (0.15.0-rc.2+)
- Install aws-cli
- Install kubectl
If you need help installing any of these applications please visit this tutorial.
Create the EKS cluster
Using eksctl we are going to create an EKS cluster in `us-east-2` region that consists of two ec2 instances as the worker nodes.
eksctl create cluster --config-file - <<EOF apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: calico-rocket region: us-east-2 version: '1.17' nodeGroups: - name: ng-calico-rocket instanceType: t3.medium desiredCapacity: 2 amiFamily: Bottlerocket iam: attachPolicyARNs: - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore EOF
Use the following command to verify bottlerocket installation on your nodes:
kubectl get nodes --output wide
You should see a similar result like:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME ip-192-168-58-69.us-east-2.compute.internal Ready <none> 110m v1.17.9 192.168.58.69 18.222.21.70 Bottlerocket OS 1.0.1 5.4.50 containerd://1.3.7+unknown ip-192-168-86-243.us-east-2.compute.internal Ready <none> 110m v1.17.9 192.168.86.243 3.137.148.137 Bottlerocket OS 1.0.1 5.4.50 containerd://1.3.7+unknown
Add Calico to the cluster
In eBPF mode, Calico implements Kubernetes Service networking directly (rather than relying on kube-proxy). This means that, like kube-proxy, Calico must connect directly to the Kubernetes API server rather than via the API server’s ClusterIP.
Execute the following command:
kubectl get configmap -n kube-system kube-proxy -o jsonpath='{.data.kubeconfig}' | grep server
You should see a result similar to the example below:
server: http://3efffae8b14fb2878c5ee75249842830.sk1.us-east-2.eks.amazonaws.com.
In this case, the <API-SERVER API>
is 3efffae8b14fb2878c5ee75249842830.sk1.us-east-2.eks.amazonaws.com
and <API-SERVER PORT>
is 443
(the standard HTTPS port). Make sure you replace these values to match your case before executing the following command.
kubectl apply -f - <<EOF kind: ConfigMap apiVersion: v1 metadata: name: kubernetes-services-endpoint namespace: kube-system data: KUBERNETES_SERVICE_HOST: "<API-SERVER API>" KUBERNETES_SERVICE_PORT: "<API-SERVER PORT>" EOF
Install Calico
Install Calico by executing the following command:
kubectl apply -f http://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/ae02a103b091f38b0aafd0ff6dd0e8f611cf9e67/config/master/calico.yaml
Note: this uses the Calico manifest from aws-cni github. (It is important to note eBPF capabilities are achievable using release 1.7.6 or higher of this manifest, so if you have an older version of the manifest, make sure you get this latest one instead.)
We can verify our Calico installation by executing the following command.
watch kubectl get pods --namespace kube-system -l k8s-app=calico-node
You should be able to see `1/1` for the `READY` column and `Running` for `STATUS` in both nodes.
NAME READY STATUS RESTARTS AGE calico-node-b2gt5 1/1 Running 0 4m5s calico-node-tgk7d 1/1 Running 0 4m5s
Use `ctrl+c` to end the watch.
Install calicoctl
Calicoctl is a multiplatform command line application that can manipulate Calico resources. There are various ways to install calicoctl. In this blog post we will install a pod version of calicoctl inside the cluster.
Note: you can safely skip this step if you already have calicoctl installed on your system. However, remember to configure it so it can communicate with your bottlerocket cluster.
kubectl apply -f http://docs.projectcalico.org/manifests/calicoctl.yaml
Create an alias for calicoctl for convenience:
alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"
Disable kube-proxy
Calico eBPF has the ability to replace `kube-proxy` pods in your cluster. By adding a node selector to `kube-proxy` pods we are going to disable them and free the resource back to the cluster.
kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "false"}}}}}'
Enable Calico eBPF mode
To enable eBPF mode, change Felix configuration parameter BPFEnabled to true. This can be done with calicoctl, as follows.
kubectl exec -i -n kube-system calicoctl -- /calicoctl patch felixconfiguration default --patch='{"spec": {"bpfEnabled": true}}'
Enabling eBPF node can disrupt existing workload connections. After enabling eBPF mode you may need to restart workload pods in order for them to restart connections.
Enable out-going NAT to internet for pods
Enable outgoing NAT for your VPC CIDR, this will allow pods to connect to the internet. (Note that in Calico 3.17 or later, you can skip this step because Calico will automatically figure out the right thing to do.)
calicoctl apply -f - <<EOF apiVersion: projectcalico.org/v3 kind: IPPool metadata: name: default-ipv4-ippool spec: cidr: 192.168.0.0/16 natOutgoing: true EOF
Congratulations, you have successfully created an EKS cluster using the Bottlerocket OS and Calico eBPF dataplane.
Clean UP
If you would like to cleanup resources created using this Blog post, simply execute the following command.
eksctl delete cluster calico-rocket
If you enjoyed this blog post then you might also like:
- Blog: Introduction the Calico eBPF dataplane
- Getting started with Calico network
- Free online training at projectcalico.org/events or subscribe to Calico Essentials for personalized training & workshops
- Learn about Calico Enterprise
Join our mailing list
Get updates on blog posts, workshops, certification programs, new releases, and more!