EKS, Bottlerocket, and Calico eBPF

Introduction

Bottlerocket is an open source Linux distribution built to run containers securely in scale by Amazon, it is uniquely tailored to improve stability and performance with a great focus on security. Utilizing a transactional update system Bottlerocket claims to reduce maintenance overhead and lower operational costs. Built from the ground up using the v5.4 Linux Kernel, makes Bottlerocket a great platform to run Calico eBPF.

In this blog post I will go through the necessary steps for setting up an EKS cluster using Bottlerocket OS and Calico eBPF. 

About Calico’s eBPF dataplane

Calico’s eBPF dataplane offers performance and capability improvements over the standard Linux networking dataplane, such as higher throughput and lower resource usage, and native Kubernetes Service handling (replacing kube-proxy) with advanced features such as source IP preservation and DSR (Direct Server Return).

If you aren’t already familiar with the concept of eBPF, it allows you to write mini programs that can be attached to various low-level hooks in the Linux kernel, for a wide variety of uses including networking, security, and tracing. You’ll see a lot of non-networking projects leveraging eBPF, but for Calico our focus is obviously on networking, and in particular, pushing the networking capabilities of the latest Linux kernel’s to the limit while maintaining Calico’s reputation for simplicity, reliability and scalability.

If you want to learn more about Calico’s eBPF dataplane, you can find much more information, including performance charts, in this blog.

Before we begin

There are few requirements that you need to have in order to follow this blogpost.

Requirements:

  • Install eksctl (0.15.0-rc.2+) 
  • Install aws-cli 
  • Install kubectl 

 If you need help installing any of these applications please visit this tutorial.

Create the EKS cluster

Using eksctl we are going to create an EKS cluster in `us-east-2` region that consists of two ec2 instances as the worker nodes. 

eksctl create cluster --config-file - <<EOF
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
 
metadata:
  name: calico-rocket
  region: us-east-2
  version: '1.17'
 
nodeGroups:
  - name: ng-calico-rocket
    instanceType: t3.medium
    desiredCapacity: 2
    amiFamily: Bottlerocket
    iam:
        attachPolicyARNs:
        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy
        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy
        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly
        - arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore
EOF

Use the following command to verify bottlerocket installation on your nodes:

kubectl get nodes --output wide

You should see a similar result like:

NAME                                           STATUS   ROLES    AGE    VERSION   INTERNAL-IP      EXTERNAL-IP     OS-IMAGE                KERNEL-VERSION   CONTAINER-RUNTIME
ip-192-168-58-69.us-east-2.compute.internal    Ready    <none>   110m   v1.17.9   192.168.58.69    18.222.21.70    Bottlerocket OS 1.0.1   5.4.50           containerd://1.3.7+unknown
ip-192-168-86-243.us-east-2.compute.internal   Ready    <none>   110m   v1.17.9   192.168.86.243   3.137.148.137   Bottlerocket OS 1.0.1   5.4.50           containerd://1.3.7+unknown

Add Calico to the cluster

In eBPF mode, Calico implements Kubernetes Service networking directly (rather than relying on kube-proxy). This means that, like kube-proxy, Calico must connect directly to the Kubernetes API server rather than via the API server’s ClusterIP.

Execute the following command:

kubectl get configmap -n kube-system kube-proxy -o jsonpath='{.data.kubeconfig}' | grep server

You should see a result similar to the example below:

   server: http://3efffae8b14fb2878c5ee75249842830.sk1.us-east-2.eks.amazonaws.com.

In this case, the <API-SERVER API> is 3efffae8b14fb2878c5ee75249842830.sk1.us-east-2.eks.amazonaws.com and <API-SERVER PORT> is 443 (the standard HTTPS port). Make sure you replace these values to match your case before executing the following command.

kubectl apply -f - <<EOF
kind: ConfigMap
apiVersion: v1
metadata:
  name: kubernetes-services-endpoint
  namespace: kube-system
data:
  KUBERNETES_SERVICE_HOST: "<API-SERVER API>"
  KUBERNETES_SERVICE_PORT: "<API-SERVER PORT>"
EOF

Install Calico

Install Calico by executing the following command:

kubectl apply -f http://raw.githubusercontent.com/aws/amazon-vpc-cni-k8s/ae02a103b091f38b0aafd0ff6dd0e8f611cf9e67/config/master/calico.yaml

Note: this uses the Calico manifest from aws-cni github. (It is important to note eBPF capabilities are achievable using release 1.7.6 or higher of this manifest, so if you have an older version of the manifest, make sure you get this latest one instead.)

We can verify our Calico installation by executing the following command.

watch kubectl get pods --namespace kube-system -l k8s-app=calico-node

You should be able to see `1/1` for the `READY` column and `Running` for `STATUS` in both nodes.

NAME            	READY   STATUS	RESTARTS   AGE
calico-node-b2gt5   1/1 	Running   0      	4m5s
calico-node-tgk7d   1/1 	Running   0      	4m5s

Use `ctrl+c` to end the watch.

Install calicoctl

Calicoctl is a multiplatform command line application that can manipulate Calico resources. There are various ways to install calicoctl. In this blog post we will install a pod version of calicoctl inside the cluster.

Note: you can safely skip this step if you already have calicoctl installed on your system. However, remember to configure it so it can communicate with your bottlerocket cluster.

kubectl apply -f http://docs.projectcalico.org/manifests/calicoctl.yaml

Create an alias for calicoctl for convenience:

alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl"

Disable kube-proxy

Calico eBPF has the ability to replace `kube-proxy` pods in your cluster. By adding a node selector to `kube-proxy` pods we are going to disable them and free the resource back to the cluster.

kubectl patch ds -n kube-system kube-proxy -p '{"spec":{"template":{"spec":{"nodeSelector":{"non-calico": "false"}}}}}'

Enable Calico eBPF mode

To enable eBPF mode, change Felix configuration parameter BPFEnabled to true. This can be done with calicoctl, as follows.

kubectl exec -i -n kube-system calicoctl -- /calicoctl patch felixconfiguration default --patch='{"spec": {"bpfEnabled": true}}'

Enabling eBPF node can disrupt existing workload connections. After enabling eBPF mode you may need to restart workload pods in order for them to restart connections.

Enable out-going NAT to internet for pods

Enable outgoing NAT for your VPC CIDR, this will allow pods to connect to the internet. (Note that in Calico 3.17 or later, you can skip this step because Calico will automatically figure out the right thing to do.)

calicoctl apply -f - <<EOF
apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: default-ipv4-ippool
spec:
  cidr: 192.168.0.0/16
  natOutgoing: true
EOF

Congratulations, you have successfully created an EKS cluster using the Bottlerocket OS and Calico eBPF dataplane. 

Clean UP

If you would like to cleanup resources created using this Blog post, simply execute the following command.

eksctl delete cluster calico-rocket

If you enjoyed this blog post then you might also like:

 

Join our mailing list

Get updates on blog posts, workshops, certification programs, new releases, and more!

X