Security Bulletins

Return to List

Summary

In certain scenarios, Calico will write configuration data in log files including service account tokens included in the configuration. This will expose client service account tokens in the log files which could lead to unauthorized cluster access or privilege escalation if users are able to access these log files. This advisory is referenced under Google Security Bulletins on November 13, 2018.

Affected Releases

• All Calico releases
  • Use of Calico on orchestrators other than Kubernetes (e.g. DC/OS) are not affected
• Tigera Secure Cloud Edition 1.0
• Tigera Secure Enterprise Edition 2.2, 2.1

Indicators of Impact/Compromise

Check your CNI configuration file; if the field “k8s_auth_token” has been set (which it is not by default in standard Calico manifests), then that token could have been written as an info level log. Otherwise, the exposure is limited to debug level logs. You can check if any service account tokens have been exposed by searching for the following log lines in the kubelet logs:

• “Updated stdin data”
• “Kubernetes config”
• “Configured environment”
• “Building client for config”
• “Config overrides”

An example using journalctl:

journalctl -u kubelet --no-pager | grep "Updated stdin data"

You can check if any service account tokens have been exposed in the calico/node logs by searching for the following log lines in the calico/node logs, for example:

kubectl logs -n kube-system -l k8s-app=calico-node -c calico-node | grep "Building client for config"

kubectl logs -n kube-system -l k8s-app=calico-node -c calico-node | grep "Config overrides"

If any of the above queries contain a Kubernetes service account token, then the cluster is affected by this vulnerability.

Audit all token usage and confirm that there is no unauthorized use of client tokens in your cluster.

Service Account Tokens generated by Kubernetes are valid so long as the token secret exists in Kubernetes. Regenerate tokens to ensure any leaked tokens no longer pose a threat.

Workaround / Remediation

The following workaround steps should be taken if you are running a version of Calico that includes this vulnerability.

1. Turn off debug logging in Calico. In the kube-system namespace:
   1. Modify the calico-config ConfigMap to set log_level to “warning” or higher.

      kubectl edit configmap -n kube-system calico-config

   2. Modify the calico-node container in the calico-node daemonset to have environment variable FELIX_LOGSEVERITYSCREEN set to “info” or higher.

      kubectl patch ds -n kube-system calico-node --patch \ '{"spec": {"template": {"spec": {"containers": [{"name": "calico-node", "env": [{"name": "FELIX_LOGSEVERITYSCREEN", "value": "warning"}]}]}}}}’

2. Delete the calico-node service account secret. In the kube-system namespace:
   1. Find the correct secret:

      kubectl get secrets -n kube-system | grep calico-node

   2. Delete the secret:

      kubectl delete secret -n kube-system

   3. A new secret will be automatically generated by the token controller.
3. Restart all calico-node pods in the kube-system namespace to pick up the new configuration.

After upgrading to a fixed version of the software, you can reset logging to whichever level you prefer.

Fixed Software

This issue has been fixed in the following patches:

• Calico v3.3.1
• Calico v3.2.4
• Calico v3.1.4
• Calico v3.0.9
• Calico v2.6.12
• Tigera Secure CE v1.0.1
• Tigera Secure EE v2.2.1

If you upgrade from an earlier version to a fixed version, if you have not already done so, you should follow steps (2) and (3) under “Workaround / Remediation” above to ensure any leaked secret is no longer valid.
Return to List