New in Calico Cloud: Continuously scan workloads in Kubernetes clusters to detect newly discovered CVEs

What is Runtime Security?

The timeline of an application can be broadly described in 3 phases:

• Build phase where the applications are in iterative development and packaged into container images.
• Deploy phase where the built container images are deployed onto staging and eventually production clusters as containers (or pods in the context of Kubernetes).
• Runtime phase where the deployed images are running actively in production as containers/pods

Thus, runtime security in the context of a cloud-native container environment broadly refers to the tools and processes leveraged to protect the operation of running containers in production.

Importance of Runtime Security

Containers are not secured by default and their configuration and environments also have a bare minimum security posture. As threats have evolved, best practices for container security need to be applied and enforced consistently. Container runtime security has become critical to get constant and up-to-date security posture of containers, especially in a Kubernetes environment, to allow teams to take action on both known vulnerabilities and anomalous behaviors of containers.

Container Image Scanning and Runtime Security

Container images are templates that are used to create new containers, and an image that contains vulnerabilities that can be exploited by outside attackers automatically passes on those vulnerabilities through the build and deploy process into the runtime environment for all containers created out of that base image.

Image Assurance is based on the Common Vulnerabilities and Exposures (CVE) system, which provides a catalog of publicly-known security vulnerabilities and exposures. Known vulnerabilities are identified by a unique CVE ID based on the year it was reported (for example, CVE-2021-44228).

Scanned image content includes:

• Libraries and content (for example, python, ruby gems, jars and go)
• Packages (OS and non-OS)
• Image layer

While container image scanning is commonly used during the build process to ensure that development teams are made aware of outdated packages and libraries that contain vulnerabilities being passed into the image, this process by itself does not protect the cluster from runtime threats in the following common scenarios:

• Images may pass scanning during the build phase, but they could contain vulnerabilities that are found days or weeks later
• Third-party images that are pulled from public registries are often not scanned in build pipelines and can contain Critical or High vulnerabilities
• Application teams may build one-off images outside of their pipeline to make an emergency patch and fix a critical bug.
• There may be existing workloads in runtime in a cluster that has no image scanning or container runtime tools implemented currently and while build-time image scanning can secure it for future images introduced to the cluster, there needs to be a way to get the security posture of already deployed image being used by running containers in such a cluster.

To remediate security gaps introduced in this way during runtime, Calico Cloud provides the user with the option of an in-cluster image scanner that runs in a Kubernetes cluster out-of-the-box ready to use without any user configuration. It runs as a daemonset in a managed cluster where images are located, and is installed on all nodes in the cluster.

Deploying the In-Cluster Scanner

The in-cluster scanner feature is disabled by default on a new managed cluster as the scanning process does consume compute due to the continuous nature of the scans.

To enable the scanner for all nodes in the cluster:

• Modify the imageassurances custom resource.

kubectl edit imageassurances default

• You will be presented with the resource YAML like so:

apiVersion: image-assurance.operator.tigera.io/v1
kind: ImageAssurance
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |                                        {"apiVersion":"image-assurance.operator.tigera.io/v1","kind":"ImageAssurance","metadata":{"annotations":{},"name":"default"},"spec":null}
  creationTimestamp: "2023-09-27T18:16:05Z"
  generation: 2
  name: default
  resourceVersion: "5486"
  uid: 26c3401b-de67-4fbd-b454-11fcf418f5da
spec:
  clusterScanner: Disabled
  containerdVolumeMountPath: /var/lib/containerd/
  criSocketPath: /run/containerd/containerd.sock

• Change the clusterScanner field value from Disabled to Enabled and save the file.
• The cluster scanner is deployed as a container inside the tigera-image-assurance-crawdad daemonset in the tigera-image-assurance namespace.

To verify, first check the pods are deployed

kubectl get pods -n tigera-image-assurance

You should see the pods deployed and running (one per worker node):

NAME                                   READY   STATUS    RESTARTS   AGE
tigera-image-assurance-crawdad-kvnd9   1/1     Running   0          41s
tigera-image-assurance-crawdad-x79pl   1/1     Running   0          41s

You can also describe one of the pod to verify the in-cluster scanner container is running:

kubectl describe pod tigera-image-assurance-crawdad-kvnd9 -n tigera-image-assurance | grep Image

Image: quay.io/tigera/image-assurance-cluster-scanner:v1.9.1
Image ID: quay.io/tigera/image-assurance-cluster-scanner@sha256:5f48a8320dc2540aabfd394f7b9577e5ec50de1c332c8d37abba7d310d66f2f7

The images will automatically start scanning.

Deploying an example workload with a vulnerable image into the cluster

As an example to understand the scanner results in the Calico Cloud UI, we will deploy an example workload that uses a vulnerable Log4j image.

• Create the following namespace and deployment

kubectl create ns java-app
kubectl create -f https://raw.githubusercontent.com/tigera-solutions/prevent-detect-and-mitigate-container-based-threats/main/apps/java-app.yaml

• Get the pods and ensure they are deployed and check the container image being used

kubectl get pods -n java-app

NAME                        READY   STATUS    RESTARTS   AGE
java-app-5b76dbc5d5-dwj5c   1/1     Running   0          44m

kubectl describe pod java-app-5b76dbc5d5-dwj5c -n java-app | grep Image

Image:          quay.io/jsabo/log4shell-vulnerable-app:latest
Image ID:   quay.io/jsabo/log4shell-vulnerable-app@sha256:6f88430688108e512f7405ac3c73d47f5c370780b94182854ea2cddc6bd59929

• Now we can check the Calico Cloud UI for the cluster scanner results about the running containers using this image by filtering by the registry/image

Clicking on the line item gives us the full CVE list that has been detected by the scanner on this image.

• The runtime image view also shows the cluster and running containers/pods using the vulnerable image

At this point, an admin can leverage the information to decide on further action to be taken with the existing workloads in a cluster.

Conclusion

Runtime security is a crucial part of strengthening the security posture of a cluster. Image scanning at build and deploy needs to be augmented by a runtime solution to detect vulnerable container images and Calico Cloud provides an in-cluster scanner that simply needs to be enabled to detect and report the results of image scans on a continuous basis and thus strengthens a cluster’s runtime security posture.

Ready to try Calico node-specific policies? Sign up for a free trial of Calico Cloud.