How to Monitor Calico’s eBPF Data Plane for Proactive Cluster Management

Monitoring is a critical part of any computer system that has been brought in to a production-ready state. No IT system exists in true isolation, and even the simplest systems interact in interesting ways with the systems “surrounding” them. Since compute time, memory, and long-term storage are all finite, it’s necessary at the very least to understand how these things are being allocated.

Why Does the Calico eBPF Data Plane Support Metrics?

Perhaps this question seems contrived. However, it’s always worth spending a moment thinking about reasons before adding any technical complexity to a distributed system! After all, they are already quite complicated! So why does the Calico eBPF data plane support metrics through Prometheus and Grafana?

Well, the Calico eBPF data plane is production ready and widely deployed, so a well-configured Kubernetes cluster with the Calico eBPF data plane correctly enabled will be stable and reliable. However, distributed systems are inherently complex and when dealing with them, it is generally good practice to instrument and baseline metrics wherever they are available. Doing so provides many benefits, especially for capacity planning, change management, and as an early-warning or smoke-testing system.

Additionally, seeing a running distributed system fully instrumented can be greatly beneficial for truly understanding how the system works and therefore how it can be improved.

The Demo Cluster

As the famous quote commonly attributed to Carl Sagan says, “If you wish to make an apple pie from scratch, you must first invent the universe.” Keeping that in mind, my objective in this blog post is to demonstrate how to gain visibility into the Calico eBPF data plane, not to demonstrate building a cluster. So, you might be glad to hear that in the interest of keeping things focused I’ll start from a running, healthy cluster with no Kubernetes CNI installed, rather than starting from bare metal.

Many clusters would be appropriate for this, but in this case my cluster is:

  • Running in Google’s GCP (not GKE)
  • Running Linux kernel version 5.11.0-1018-gcp
  • Built with kubeadm
  • No CNI, yet

Here’s what it will look like:

Example K8s Cluster in Google GCEExamining the cluster shows that it is in a NotReady state, which is to be expected since it does not yet have a CNI.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─492─▶ kubectl get nodes -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
chris-t-controller NotReady control-plane,master 18h v1.21.4 10.240.0.11 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7
chris-t-worker-0 NotReady <none> 17h v1.21.4 10.240.0.20 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7
chris-t-worker-1 NotReady <none> 17h v1.21.4 10.240.0.21 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7
chris-t-worker-2 NotReady <none> 17h v1.21.4 10.240.0.22 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─493─▶ kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-558bd4d5db-bvkml 0/1 Pending 0 18h
kube-system coredns-558bd4d5db-nl4s9 0/1 Pending 0 18h
kube-system etcd-chris-t-controller 1/1 Running 0 18h
kube-system kube-apiserver-chris-t-controller 1/1 Running 0 18h
kube-system kube-controller-manager-chris-t-controller 1/1 Running 0 18h
kube-system kube-proxy-f8sd7 1/1 Running 0 18h
kube-system kube-proxy-fvz6p 1/1 Running 0 17h
kube-system kube-proxy-grwzq 1/1 Running 0 17h
kube-system kube-proxy-t77zs 1/1 Running 0 17h
kube-system kube-scheduler-chris-t-controller 1/1 Running 0 18h

First, I apply the Tigera Operator, which is an operator following the Operator pattern that is responsible for deploying Calico Open Source on the cluster.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─494─▶ kubectl create -f http://docs.projectcalico.org/manifests/tigera-operator.yaml
customresourcedefinition.apiextensions.k8s.io/bgpconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/bgppeers.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/blockaffinities.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/caliconodestatuses.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/clusterinformations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/felixconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/globalnetworksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/hostendpoints.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamblocks.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamconfigs.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipamhandles.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ippools.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/ipreservations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/kubecontrollersconfigurations.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networkpolicies.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/networksets.crd.projectcalico.org created
customresourcedefinition.apiextensions.k8s.io/apiservers.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/imagesets.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/installations.operator.tigera.io created
customresourcedefinition.apiextensions.k8s.io/tigerastatuses.operator.tigera.io created
namespace/tigera-operator created
Warning: policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
podsecuritypolicy.policy/tigera-operator created
serviceaccount/tigera-operator created
clusterrole.rbac.authorization.k8s.io/tigera-operator created
clusterrolebinding.rbac.authorization.k8s.io/tigera-operator created
deployment.apps/tigera-operator created

Next, I create a file defining the Installation and APIServer resources, and apply it to the cluster. In fact, this is just the normal Calico install process documented on the Calico Open Source docs site. The only change is that I have specified a typhaMetricsPort. As noted in the Installation Reference, this specifies which port Calico’s Typha component serves Prometheus metrics on. By default, this is not enabled.

chris @ chris-work ~/2021/2021_11/metrics_demo
└─500─▶ cat << EOF > chris-t-custom-resources.yaml
cat << EOF > chris-t-custom-resources.yaml
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
name: default
spec:
calicoNetwork:
ipPools:
- blockSize: 26
cidr: 192.168.0.0/16
encapsulation: VXLANCrossSubnet
natOutgoing: Enabled
nodeSelector: all()
typhaMetricsPort: 9093
---
apiVersion: operator.tigera.io/v1
kind: APIServer
metadata:
name: default
spec: {}
EOF

chris @ chris-work ~/2021/2021_11/metrics_demo
└─501─▶ kubectl create -f chris-t-custom-resources.yaml
installation.operator.tigera.io/default created
apiserver.operator.tigera.io/default created

Well, that was easy! At this point, the cluster is running Calico. However, I want to use the eBPF data plane, so now I need to enable that. I won’t show the steps in detail here since I did not deviate at all from the documented process for enabling the eBPF data plane on a Calico cluster. At a high level, I:

  • Created a ConfigMap resource instructing Calico to talk directly to the Kubernetes API rather than via kube-proxy
  • Instructed the cluster to disable kube-proxy
  • Changed the Installation resource to instruct the Tigera Operator to enable the eBPF data plane

If you want to know more about those steps, remember that besides the documentation above, there are lots of experienced community members in the #eBPF channel on the Calico Users Slack who can help.

Understanding the Available Targets

You probably don’t really need to know exactly what metrics look like on the various targets in a Kubernetes cluster using Calico’s eBPF data plane, but it is often a good idea to dive into the details a little to get some insights. Metrics in this kind of cluster are available from:

  • Typha (Calico’s datastore fan-out daemon)
  • calico-kube-controllers (Calico’s container for the controllers that perform actions based on cluster state)
  • Felix (Calico’s per-node agent)

Understanding Discovery Services

One subtlety of monitoring all three of the above targets is how Kubernetes services are used. They are not used in the traditional way! I’ll give the example of Felix, but it relates to all of the available targets.

This is perhaps best explained with a diagram. As shown in the black lines in the diagram below, services in Kubernetes are usually used to forward traffic to a set of service pods. However, consider the case of Felix metrics collection. Each Felix pod has different metrics, and I want Prometheus to collect them all. If it was load-balanced by the service, that would result in each request for metrics being balanced to a different pod, and data being missed or muddled.

FelixDiscoveryViaServiceExample.drawio.png

Therefore, the service is used for discovery only. When Prometheus wants to know where the metrics endpoints are, it does query the service, but the actual polling traffic is always sent directly to the target pod, never via the service.

This use of a service is interesting and non-intuitive. This discovery mode is called the endpoints role, and is in this Prometheus documentation. I wondered if it might have been better to use a different method for this kind of discovery. After a little investigation, I believe that at the time that this Calico functionality was originally designed, the endpoints role was recommended, though Prometheus now offers an additional pod role that could be used as an alternative if this functionality is being improved or reimplemented.

For the time being though, this is a simple and feature-complete implementation that conforms to the Prometheus documentation.

Typha’s Metrics

To view Typha’s metrics, I first grab the IP of one of the nodes running a Typha pod, and then curl port 9093 from within the cluster.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─518─▶ kubectl get pods -A -o wide | grep -i typha
calico-system calico-typha-79db7f5747-8m4sv 1/1 Running 0 3h32m 10.240.0.20 chris-t-worker-0 <none> <none>
calico-system calico-typha-79db7f5747-qtlsk 1/1 Running 0 3h32m 10.240.0.22 chris-t-worker-2 <none> <none>


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─519─▶ gcloud compute ssh ubuntu@chris-t-controller --zone us-central1-a --command "curl -s http://10.240.0.20:9093/metrics" | head -n 10
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.1838e-05
go_gc_duration_seconds{quantile="0.25"} 0.000117127
go_gc_duration_seconds{quantile="0.5"} 0.000134296
go_gc_duration_seconds{quantile="0.75"} 0.000169608
go_gc_duration_seconds{quantile="1"} 0.00040328
go_gc_duration_seconds_sum 0.017567578
go_gc_duration_seconds_count 116
# HELP go_goroutines Number of goroutines that currently exist.

The output above shows that the Typha pod is correctly responding on port 9093 as I configured.

Next, I create the discovery service that Prometheus will use, and grab its IP.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─520─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: typha-metrics-svc
namespace: calico-system
spec:
selector:
k8s-app: calico-typha
ports:
- port: 9093
targetPort: 9093
EOF

service/typha-metrics-svc created


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─521─▶ kubectl get services -n=calico-system typha-metrics-svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
typha-metrics-svc ClusterIP 10.100.83.54 <none> 9093/TCP 16s

I will test the service to ensure it’s responding, but keep in mind the caveat in the previous section: this service is not actually polled in this way once Prometheus is running.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─522─▶ gcloud compute ssh ubuntu@chris-t-controller --zone us-central1-a --command "curl -s http://10.100.83.54:9093/metrics" | head -n 10
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 2.1838e-05
go_gc_duration_seconds{quantile="0.25"} 0.000117127
go_gc_duration_seconds{quantile="0.5"} 0.000135733
go_gc_duration_seconds{quantile="0.75"} 0.000169189
go_gc_duration_seconds{quantile="1"} 0.00040328
go_gc_duration_seconds_sum 0.017886049
go_gc_duration_seconds_count 118
# HELP go_goroutines Number of goroutines that currently exist.

That’s it, for now! Once we add Prometheus to the cluster, we will pass it the name of this service to enable discovery.

calico-kube-controllers’ Metrics

The process for calico-kube-controllers is very similar. First, I confirm the configured port. In this case, it’s the default port 9094. Then I grab the pod IP and test from within the cluster, as before.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─527─▶ kubectl get kubecontrollersconfiguration default -o yaml | grep -i prometheusMetricsPort
prometheusMetricsPort: 9094


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─528─▶ kubectl get pods -A -o wide | grep -i calico-kube-controllers
calico-system calico-kube-controllers-669995d4df-q9sd8 1/1 Running 0 3h57m 192.168.203.4 chris-t-worker-1 <none> <none>


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─529─▶ gcloud compute ssh ubuntu@chris-t-controller --zone us-central1-a --command "curl -s http://192.168.203.4:9094/metrics" | head -n 10# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.5216e-05
go_gc_duration_seconds{quantile="0.25"} 0.000153376
go_gc_duration_seconds{quantile="0.5"} 0.000180758
go_gc_duration_seconds{quantile="0.75"} 0.000216069
go_gc_duration_seconds{quantile="1"} 0.002992162
go_gc_duration_seconds_sum 0.028429418
go_gc_duration_seconds_count 128
# HELP go_goroutines Number of goroutines that currently exist.

Next, as before, I create the discovery service that Prometheus will use, and grab its IP.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─530─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: kube-controllers-metrics-svc
namespace: calico-system
spec:
selector:
k8s-app: calico-kube-controllers
ports:
- port: 9094
targetPort: 9094
EOF
service/kube-controllers-metrics-svc created


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─531─▶ kubectl get services -n=calico-system kube-controllers-metrics-svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-controllers-metrics-svc ClusterIP 10.98.85.108 <none> 9094/TCP 25s

Again, I will test the service to ensure it’s responding, with the same caveat as before.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─533─▶ gcloud compute ssh ubuntu@chris-t-controller --zone us-central1-a --command "curl -s http://10.98.85.108:9094/metrics" | head -n 10
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.5216e-05
go_gc_duration_seconds{quantile="0.25"} 0.000153376
go_gc_duration_seconds{quantile="0.5"} 0.000180758
go_gc_duration_seconds{quantile="0.75"} 0.000216069
go_gc_duration_seconds{quantile="1"} 0.002992162
go_gc_duration_seconds_sum 0.029002837
go_gc_duration_seconds_count 131
# HELP go_goroutines Number of goroutines that currently exist.

That’s it! Again, once we add Prometheus, we will pass it the name of this service to enable discovery.

Felix’s metrics

Finally, let’s repeat the process for Felix. This time, I enable metrics reporting by setting prometheusMetricsEnabled to true in the default FelixConfiguration resource. Then I grab the node IP and test from within the cluster, as before.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─535─▶ kubectl patch felixConfiguration default --patch '{"spec":{"prometheusMetricsEnabled": true}}'
Successfully patched 1 'FelixConfiguration' resource


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─536─▶ kubectl get felixConfiguration default -o yaml
apiVersion: projectcalico.org/v3
kind: FelixConfiguration
metadata:
creationTimestamp: "2022-01-05T09:51:31Z"
name: default
resourceVersion: "144928"
uid: 17977eaf-7324-4da1-a4a0-54e46c5f0b93
spec:
bpfLogLevel: ""
logSeverityScreen: Info
prometheusMetricsEnabled: true
reportingInterval: 0s
vxlanEnabled: true


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─537─▶ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
chris-t-controller Ready control-plane,master 23h v1.21.4 10.240.0.11 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7
chris-t-worker-0 Ready <none> 22h v1.21.4 10.240.0.20 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7
chris-t-worker-1 Ready <none> 22h v1.21.4 10.240.0.21 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7
chris-t-worker-2 Ready <none> 22h v1.21.4 10.240.0.22 <none> Ubuntu 20.04.3 LTS 5.11.0-1018-gcp docker://20.10.7


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─538─▶ gcloud compute ssh ubuntu@chris-t-controller --zone us-central1-a --command "curl -s http://10.240.0.20:9091/metrics" | head -n 10
# HELP felix_active_local_endpoints Number of active endpoints on this host.
# TYPE felix_active_local_endpoints gauge
felix_active_local_endpoints 1
# HELP felix_active_local_policies Number of active policies on this host.
# TYPE felix_active_local_policies gauge
felix_active_local_policies 0
# HELP felix_active_local_selectors Number of active selectors on this host.
# TYPE felix_active_local_selectors gauge
felix_active_local_selectors 0
# HELP felix_active_local_tags Number of active tags on this host.

Next, as before, I create the discovery service that Prometheus will use, and grab its IP.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─539─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: felix-metrics-svc
namespace: calico-system
spec:
selector:
k8s-app: calico-node
ports:
- port: 9091
targetPort: 9091
EOF

service/felix-metrics-svc created


chris @ chris-work ~/2021/2021_11/metrics_demo 
└─540─▶ kubectl get services -n=calico-system felix-metrics-svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
felix-metrics-svc ClusterIP 10.99.145.23 <none> 9091/TCP 7s

Again, I will test the service to ensure it’s responding, with the same caveat as before.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─542─▶ gcloud compute ssh ubuntu@chris-t-controller --zone us-central1-a --command "curl -s http://10.99.145.23:9091/metrics | head -n 10"
# HELP felix_active_local_endpoints Number of active endpoints on this host.
# TYPE felix_active_local_endpoints gauge
felix_active_local_endpoints 2
# HELP felix_active_local_policies Number of active policies on this host.
# TYPE felix_active_local_policies gauge
felix_active_local_policies 1
# HELP felix_active_local_selectors Number of active selectors on this host.
# TYPE felix_active_local_selectors gauge
felix_active_local_selectors 0
# HELP felix_active_local_tags Number of active tags on this host.

That’s it! Again, once we add Prometheus, we will pass it the name of this service to enable discovery.

Enable Prometheus on the cluster

Up until this point, I have been enabling metrics endpoints on monitorable services in the Calico-eBPF-data-plane Kubernetes cluster. This does not mean that any statistics are actually being collected! All I have done so far is enable the HTTP targets that make the data available for Prometheus to collect.

In order to actually start collecting the statistics, I need to install the central Prometheus instance (pod) that collects the data in a time-series database.

First, I need to create a namespace to host it.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─543─▶ kubectl apply -f -<<EOF
apiVersion: v1
kind: Namespace
metadata:
name: calico-monitoring
labels:
app: ns-calico-monitoring
role: monitoring
EOF

namespace/calico-monitoring created

Next, I need to create a ClusterRole, which represents the actions and resources I want to allow the Prometheus service to access. After that, I must create a ServiceAccount for the Prometheus service, and bind the two using a ClusterRoleBinding.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─545─▶ kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: calico-prometheus-user
rules:
- apiGroups: [""]
resources:
- endpoints
- services
- pods
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: calico-prometheus-user
namespace: calico-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: calico-prometheus-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: calico-prometheus-user
subjects:
- kind: ServiceAccount
name: calico-prometheus-user
namespace: calico-monitoring
EOF

clusterrole.rbac.authorization.k8s.io/calico-prometheus-user created
serviceaccount/calico-prometheus-user created
clusterrolebinding.rbac.authorization.k8s.io/calico-prometheus-user created

After that, I need to create a configuration file for Prometheus, and store it in a ConfigMap, as is commonplace for providing configuration files to Kubernetes-based services. In particular, notice how the three kubernetes_sd_configs are included, with role: endpoints statements, and each points to one of the services we configured and tested. This is how Prometheus will discover the endpoints.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─546─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: prometheus-config
namespace: calico-monitoring
data:
prometheus.yml: |-
global:
scrape_interval: 15s
external_labels:
monitor: 'tutorial-monitor'
scrape_configs:
- job_name: 'prometheus'
scrape_interval: 5s
static_configs:
- targets: ['localhost:9090']
- job_name: 'felix_metrics'
scrape_interval: 5s
scheme: http
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
regex: felix-metrics-svc
replacement: $1
action: keep
- job_name: 'typha_metrics'
scrape_interval: 5s
scheme: http
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
regex: typha-metrics-svc
replacement: $1
action: keep
- job_name: 'kube_controllers_metrics'
scrape_interval: 5s
scheme: http
kubernetes_sd_configs:
- role: endpoints
relabel_configs:
- source_labels: [__meta_kubernetes_service_name]
regex: calico-kube-controllers-metrics
replacement: $1
action: keep
EOF
configmap/prometheus-config created

Now that the targets and services are configured, and Prometheus has a configuration file stored in the ConfigMap, it’s time that I create the pod that will load the configuration and actually start doing the statistics collection work. Doing so is just a case of creating a Pod Kubernetes resource, which actually specifies what image should run and with what resources and configuration, and a Service Kubernetes resource that presents the new Prometheus pod as a service consumable by anything else in the cluster.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─487─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: prometheus-pod
namespace: calico-monitoring
labels:
app: prometheus-pod
role: monitoring
spec:
serviceAccountName: calico-prometheus-user
containers:
- name: prometheus-pod
image: prom/prometheus
resources:
limits:
memory: "128Mi"
cpu: "500m"
volumeMounts:
- name: config-volume
mountPath: /etc/prometheus/prometheus.yml
subPath: prometheus.yml
ports:
- containerPort: 9090
volumes:
- name: config-volume
configMap:
name: prometheus-config
EOF

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─488─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
name: prometheus-dashboard-svc
namespace: calico-monitoring
spec:
selector:
app: prometheus-pod
role: monitoring
ports:
- port: 9090
targetPort: 9090
EOF
service/prometheus-dashboard-svc created

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─489─▶ kubectl port-forward service/prometheus-dashboard-svc 9090:9090 -n calico-monitoring
Forwarding from 127.0.0.1:9090 -> 9090
Forwarding from [::1]:9090 -> 9090

That’s it! Prometheus is running.

Screenshot of a Prometheus dashboardAt this point I can query the data, but Prometheus is usually used with Grafana as a visualization frontend.

Enable Grafana

Luckily, setting up Grafana is easy. First, I just add a config file in a ConfigMap as I did for Prometheus. The configuration is minimal; it just needs to instruct Grafana where to find the Prometheus service I set up.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─491─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
name: grafana-config
namespace: calico-monitoring
data:
prometheus.yaml: |-
{
"apiVersion": 1,
"datasources": [
{
"access":"proxy",
"editable": true,
"name": "calico-demo-prometheus",
"orgId": 1,
"type": "prometheus",
"url": "http://prometheus-dashboard-svc.calico-monitoring.svc:9090",
"version": 1
}
]
}
EOF
configmap/grafana-config created

Then, I create a ConfigMap with some stock Felix and Typha dashboards.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─492─▶ kubectl apply -f http://docs.projectcalico.org/manifests/grafana-dashboards.yaml
configmap/grafana-dashboards-config created

Next, I create the Grafana pod.

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─493─▶ kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
name: grafana-pod
namespace: calico-monitoring
labels:
app: grafana-pod
role: monitoring
spec:
containers:
- name: grafana-pod
image: grafana/grafana:latest
resources:
limits:
memory: "128Mi"
cpu: "500m"
volumeMounts:
- name: grafana-config-volume
mountPath: /etc/grafana/provisioning/datasources
- name: grafana-dashboards-volume
mountPath: /etc/grafana/provisioning/dashboards
- name: grafana-storage-volume
mountPath: /var/lib/grafana
ports:
- containerPort: 3000
volumes:
- name: grafana-storage-volume
emptyDir: {}
- name: grafana-config-volume
configMap:
name: grafana-config
- name: grafana-dashboards-volume
configMap:
name: grafana-dashboards-config
EOF
pod/grafana-pod created

chris @ chris-work ~/2021/2021_11/metrics_demo 
└─494─▶ kubectl port-forward pod/grafana-pod 3000:3000 -n calico-monitoring
Forwarding from 127.0.0.1:3000 -> 3000
Forwarding from [::1]:3000 -> 3000

And that’s it, Grafana is running! The default username and password is admin/admin, so I can log in and start creating dashboards and visualizations.

Screenshot showing a Grafana DashboardNext steps

At this point, you can explore, learn about, and create visualizations and dashboards for the available eBPF data plane metrics that are now being ingested by Prometheus. In a follow-on blog post, we will examine the available metrics and what can be learned from them.

Did you know you can become a certified Calico operator? Learn container and Kubernetes networking and security fundamentals using Calico in this free, self-paced certification course.
There are additional level two courses as well. One of them specifically addresses eBPF and the Calico eBPF data plane!

Join our mailing list

Get updates on blog posts, new releases and more!