Your Step-by-Step to Observability Without the Operator
If you’re running Calico using manifests, you may have found that enabling the observability features introduced in version 3.30, including Calico Whisker `and Goldmane, requires a more hands-on approach. Earlier documentation focused on the Tigera operator, which automates key tasks such as certificate management and secure service configuration. In a manifest-based setup, these responsibilities shift to the user. While the process involves more manual steps, it provides greater transparency and control over each component. With the right guidance, setting up these observability tools is entirely achievable and offers valuable insight into the internal behavior of your Calico deployment.
We’ve heard from many of you in the Calico Slack community: you’re eager to try out Calico Whisker and Goldmane but aren’t sure how to set them up without Helm or the operator. For anyone who’s up for a challenge, this blog post provides a step-by-step guide on how to get everything wired up the hard way.
However, even if you already use the operator, keep reading! We’re going to pull back the curtain on the magic it performs behind the scenes. Understanding these mechanics will help you troubleshoot, customize, and better appreciate a managed approach, whether you’re an SRE, platform engineer, or a curious cluster admin.
What you will achieve at the end of this blog:
- You will learn how to get up and running with Calico Whisker and Goldmane in a manifest based installation.
- You will learn why a Helm chart or the Tigera operator is our recommendation to manage your Kubernetes workloads at scale.
- You will have some hands on experience on what to look for when things are deployed in a real life scenario.
Why Calico Whisker and Goldmane Matter
Without Goldmane and tools, you’re essentially flying blind and you:
- Can’t easily debug why traffic is dropped
- Can’t visualize how policies affect flows
- Can’t trace service-to-service communication paths
This leads to increased time-to-resolution for network bugs, frustration from teams consuming the platform, limited audit capabilities for security teams
For manifest-only users, the lack of clear installation steps means many miss out on these benefits and as a developer advocate, I’m here to fix that!
Installing Calico Whisker and Goldmane Manually
Requirements
If you want to replicate this setup in your own environment, you’ll need to install or have a few key requirements in place:
- A manifest based installation of Calico v3.30.2 and Calico Typha (EKS, AKS, GKE, on-prem)
- OpenSSL (To issue and sign certificates, operator install sounds better and better right?)
Please keep in mind that this blog post assumes youare running Calico v3.30.2, if you version differs you might have to adjust some of the steps.
When installing Calico with manifests, you’re responsible for configuring every component manually. While this gives you maximum flexibility, it also comes with notable risks:
- YAML formatting errors: Simple typos can break your configuration.
- Using deprecated or unsupported settings: You might miss critical updates, leading to instability.
- Forgetting to carry over custom changes during upgrades: Your manual customizations can get lost, causing unexpected behavior.
These challenges often lead to time-consuming and frustrating troubleshooting. That’s why we recommend the tigera-operator, which automates configuration and upgrades while still allowing for a high degree of customization.
The operator is a dedicated tool, tuned to configure and maintain your Calico installation so you can spend your time on the things that matter most—like vacations!
Generating Certificates and Certificate Authority (CA)
By default, the manifest installation does not secure Calico components. It is your responsibility as the administrator to generate a Certificate Authority and certificates, then sign and rotate them on your desired schedule. Keep in mind that all of these steps are automated with an operator-based installation. Yes, the list is a step-by-step guide. It breaks down the process of manually preparing and importing certificates for Calico’s observability features.
The list can be made even clearer for a better step-by-step experience by grouping related actions and using consistent language:
Step-by-Step Certificate Generation
- Create your Certificate Authority (CA): Establish a root of trust for your certificates.
- Generate Certificate Signing Requests (CSRs): Create requests for Calico-Typha, Calico-Node, Goldmane, and Calico Whisker. Ensure the Goldmane and Calico Whisker requests specify the correct key exchange role and Subject Alternative Names (SANs).
- Sign the Certificates: Use your CA key to sign all the generated CSRs.
- Import Certificates and Keys into your environment:
- Import the CA certificate as a ConfigMap named goldmane in the kube-system namespace.
- Import the typha-client and typha-server certs and keys as TLS secrets named typha-client-key-pair and typha-server-key-pair.
- Import the goldmane key and cert as a TLS secret named goldmane-key-pair.
- Import the Calico Whisker key and cert as a TLS secret named whisker-backend-key-pair
Need help generating the certificates? Click here to learn more.
Tuning Typha to use certificates
Now that the certification “ritual” is complete, the next step is to assign these certificates to your Calico components. Let’s start with Typha.
To add certificates to Typha, you’ll need to modify the calico-typha deployment. You’ll add a VolumeMount object to connect the secrets you generated to the pods, define a volume to mount them within the workloads, and set a few environment variables to let Typha know it’s time to use certificates.
Use the following command to modify your Typha deployment:
kubectl patch deployment calico-typha -n kube-system --type=strategic -p='
{
"spec": {
"template": {
"spec": {
"volumes": [
{
"name": "config",
"configMap": {
"name": "goldmane",
"defaultMode": 420
}
},
{
"name": "goldmane-ca-bundle",
"configMap": {
"name": "goldmane-ca-bundle",
"defaultMode": 420
}
},
{
"name": "typha-server-key-pair",
"secret": {
"secretName": "typha-server-key-pair",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "calico-typha",
"env": [
{
"name": "TYPHA_CAFILE",
"value": "/etc/pki/tls/certs/tigera-ca-bundle.crt"
},
{
"name": "TYPHA_SERVERCERTFILE",
"value": "/typha-server-key-pair/tls.crt"
},
{
"name": "TYPHA_SERVERKEYFILE",
"value": "/typha-server-key-pair/tls.key"
},
{
"name": "TYPHA_CLIENTCN",
"value": "typha-client"
}
],
"volumeMounts": [
{
"name": "goldmane-ca-bundle",
"mountPath": "/etc/pki/tls/certs",
"readOnly": true
},
{
"name": "typha-server-key-pair",
"mountPath": "/typha-server-key-pair",
"readOnly": true
}
]
}
]
}
}
}
}'Note: We’ve shared the CA certificate (not the key) with the cluster using the TYPHA_CAFILE environment variable, since we’re using a private CA. This lets Typha validate client certs.
Next, patch the calico-node DaemonSet to use our certificates and establish mTLS:
- Set FELIX_TYPHACN=typha-server to match the Typha server certificate’s Common Name.
- This enables calico-node to verify Typha’s identity.
Fixing DNS Issues
Because calico-node runs with hostNetwork: true, it uses the host’s DNS, which can’t resolve internal Kubernetes service names like goldmane.kube-system.svc.
To fix this, we need to set the dnsPolicy to ClusterFirstWithHostNet so it uses the cluster DNS server:
kubectl patch ds calico-node -n kube-system --type=strategic -p='
{
"spec": {
"template": {
"spec": {
"dnsPolicy": "ClusterFirstWithHostNet",
"volumes": [
{
"name": "config",
"configMap": {
"name": "goldmane",
"defaultMode": 420
}
},
{
"name": "goldmane-ca-bundle",
"configMap": {
"name": "goldmane-ca-bundle",
"defaultMode": 420
}
},
{
"name": "typha-client-key-pair",
"secret": {
"secretName": "typha-client-key-pair",
"defaultMode": 420
}
}
],
"containers": [
{
"name": "calico-node",
"env": [
{
"name": "FELIX_TYPHACAFILE",
"value": "/etc/pki/tls/certs/tigera-ca-bundle.crt"
},
{
"name": "FELIX_TYPHACERTFILE",
"value": "/typha-client-key-pair/tls.crt"
},
{
"name": "FELIX_TYPHAKEYFILE",
"value": "/typha-client-key-pair/tls.key"
},
{
"name": "FELIX_TYPHAK8SNAMESPACE",
"value": "kube-system"
},
{
"name": "FELIX_TYPHACN",
"value": "typha-server"
},
{
"name": "FELIX_FLOWLOGSGOLDMANESERVER",
"value": "goldmane.kube-system.svc:7443"
},
{
"name": "FELIX_FLOWLOGSFLUSHINTERVAL",
"value": "15"
}
],
"volumeMounts": [
{
"name": "goldmane-ca-bundle",
"mountPath": "/etc/pki/tls/certs",
"readOnly": true
},
{
"name": "typha-client-key-pair",
"mountPath": "/typha-client-key-pair",
"readOnly": true
}
]
}
]
}
}
}
}'Before you begin the next step, use the following command to ensure that Calico and Typha have applied your changes.
kubectl rollout status -n kube-system ds/calico-node deployment/calico-typhaDeploying Goldmane
Now that your Calico and Typha workloads are running and secured, let’s deploy Goldmane and Calico Whisker.
First, you’ll need to create a Service Account. Service accounts provide an identity for workloads, allowing administrators to manage their permissions.
Use the following command to generate a service account for Goldmane:
kubectl create -f -<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: goldmane
namespace: kube-system
EOFNext, you need to create a service that points to the Goldmane ingestion port. This service will be used by any emitter (like calico-node) to send flow data to the Goldmane server.
Use the following command to generate the service:
kubectl create -f -<<EOF
apiVersion: v1
kind: Service
metadata:
name: goldmane
namespace: kube-system
spec:
ports:
- port: 7443
protocol: TCP
targetPort: 7443
selector:
k8s-app: goldmane
sessionAffinity: None
type: ClusterIP
EOFThe Goldmane binary looks for a config file to be present, so you need to supply that file to Goldmane when it starts.
Use the following command to generate the config file as a ConfigMap.
kubectl create -f -<<EOF
apiVersion: v1
data:
config.json: '{"emitFlows":false}'
kind: ConfigMap
metadata:
name: goldmane
namespace: kube-system
EOFTuning Goldmane Deployment
The Goldmane binary looks for specific settings to adjust its behavior. Since certificates have a finite lifespan, we’ll use environment variables to tell Goldmane which ones to use.
Review the following block, as it shows the necessary changes to the Goldmane deployment that you will apply later.
- name: SERVER_CERT_PATH
value: /goldmane-key-pair/tls.crt
- name: SERVER_KEY_PATH
value: /goldmane-key-pair/tls.key
- name: CA_CERT_PATH
value: /etc/pki/tls/certs/tigera-ca-bundle.crtSimilar to the previous steps, you’ll need to mount the VolumeMounts to the workload so they can be accessed from within the container.
volumes:
- configMap:
defaultMode: 420
name: goldmane
name: config
- configMap:
defaultMode: 420
name: goldmane-ca-bundle
name: goldmane-ca-bundle
- name: goldmane-key-pair
secret:
defaultMode: 420
secretName: goldmane-key-pairFinally, we need to mount these VolumeMounts in their respective path inside the workload.
volumeMounts:
- mountPath: /config
name: config
readOnly: true
- mountPath: /etc/pki/tls/certs
name: goldmane-ca-bundle
readOnly: true
- mountPath: /goldmane-key-pair
name: goldmane-key-pair
readOnly: trueNote: The following gist file has all the changes that we examined in the above.
Use the following command to deploy Goldmane:
kubectl create -f https://gist.githubusercontent.com/frozenprocess/5555ec3266133e53510b4bda59f38e42/raw/003335e822557b8bd56b0180418ec1bf6f7232cb/goldmane.yaml
Deploying Calico Whisker UI
It’s time to deploy the Calico Whisker UI. We will follow the same process as Goldmane to create a service account, a service, and configure the deployment to use certificates.
Use the following command to generate the service account and service:
kubectl create -f -<<EOF
apiVersion: v1
kind: ServiceAccount
metadata:
name: whisker
namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
name: whisker
namespace: kube-system
spec:
ports:
- port: 8081
protocol: TCP
targetPort: 8081
selector:
k8s-app: whisker
sessionAffinity: None
type: ClusterIP
EOFTuning Calico Whisker Deployment
Similar to Goldmane binary, Calico Whisker looks for certain indications to change its behavior, since certificates have a finite lifetime, we are using environment variables to tell Calico Whisker which certificates it should use. On top of that Calico Whisker needs a URL to know where Goldmane resides; this is achieved by setting the GOLDMANE_HOST environment variable in the deployment.
Examine the following block, it is an example that illustrates the required unique changes in Calico Whisker deployment that you will deploy later.
- env:
- name: LOG_LEVEL
value: INFO
- name: PORT
value: "3002"
- name: GOLDMANE_HOST
value: goldmane.kube-system.svc.cluster.local:7443Note: The following gist file has all the changes that we examined in the above.
Use the following command to deploy Calico Whisker:
kubectl create -f https://gist.githubusercontent.com/frozenprocess/5555ec3266133e53510b4bda59f38e42/raw/003335e822557b8bd56b0180418ec1bf6f7232cb/whisker.yamlYou can verify Goldmane and Calico Whisker deployment in a Manifest based install by using the following command:
kubectl rollout status -n kube-system deployment/goldmane
kubectl rollout status -n kube-system deployment/whiskerAccessing Calico Whisker
Note: The following step needs to be running if you wish to access Calico Whisker UI using a ClusterIP approach.
Open a new terminal and run the following command:
kubectl port-forward -n kube-system deployment/whisker 8081:8081Now open a browser and point it to localhost:8081, you should see the Calico Whisker UI and your flows should appear shortly.
Observability Unlocked (and Why the Operator Helps)
With this setup in place, you now have:
- ✅ Calico Whisker UI: A real-time service graph and policy flow debugger
- ✅ Goldmane backend: Ingests flow logs from Felix or anywhere else, powers Calico Whisker
- ✅ No Helm, no Operator: Just manifests.
But as you probably noticed, this took a lot of effort. In comparison, the Tigera operator automates all of this for you:
- It automatically deploys and wires Goldmane and Calico Whisker.
- It enables mTLS out of the box.
- It adds health checks and service account permissions.
- It automatically responds to CRD-based config changes.
- It provides subcommands like tigerastatus to verify your installation.
- It maintains Calico and all its components so you don’t have to worry.
While manifests work, using the operator saves you time, reduces errors, and helps future-proof your setup.
Final Thoughts
Whether you’re a minimalist or a control freak (we see you), it’s great to understand how Calico observability tools work under the hood. But when it comes to production and saving engineering time, the operator can help take care of the wiring so you can enjoy your vacations.
💬 Have questions or want to contribute improvements to the manifest flow? Join the discussion on Calico Slack. Click here to learn more about what is possible with Calico Whisker.
