Preventing Vulnerable Container Deployments with Admission Control

In a previous blog post, Hands-on guide: How to scan and block container images to mitigate SBOM attacks, we looked at how Software Supply Chain threats can be identified and assessed. The severity of these vulnerabilities determine the posture or scan result for an image i.e. Pass, Warning or Fail. The next question is “What can we do with these results?”. To improve the security posture to reduce attacks on your workload we must ensure that workloads have the fewest possible vulnerabilities and layer on configuration security with KSPM, egress controls, and microsegmentation.

In this post we will cover how the scan results can be leveraged to add an additional layer of protection during Deploy Time in application deployment lifecycles.

It’s worth noting that Calico’s Image Scanner is an offline binary which can be run locally. This means the Image Scanner can be baked into any existing Continuous Integration/Continuous Delivery(CI/CD) pipeline. For example, after an image has been built the image can be scanned by the Image Scanner in an Execution Environment. Here checks can be configured to prevent the image from being pushed to a registry should vulnerabilities be detected. This is effectively how image scanning can be utilized during build time.

Deploy Time Protection

Even when image scanning is performed during build time there are scenarios when it is appropriate to secure the deployment of container images during deploy time:

• There is a time delay from when an image is built to when it is deployed. New vulnerabilities may be detected in between these phases.
• Not all images have gone through a controlled build procedure in a CI/CD pipeline. For example, images which have been built externally.
• Deployment of external images for ancillary services such as logging, analysis, visibility, troubleshooting and/or additional supporting capabilities.

Therefore, it is prudent for environments which require the strictest of security controls that the images must be scanned prior to deployment. Calico makes use of Kubernetes’ Dynamic Admission Control to provide a gatekeeper function to police the images which can be deployed into a cluster. Calico’s Admission Controller is a resource that operates at the cluster level. It relies on a validating admission webhook whereby the Kubernetes API checks with the Admission Controller if a specific Kubernetes resource, based on a specific image, can be created or updated. But how does the Admission Controller know whether to accept or deny a request?

Enter Container Admission policies. The Admission Controller refers to Container Admission Policies in order to allow or reject requests made towards the Kubernetes API for resource (e.g. pods and deployments) creation or updates. Container Admission Policies are Kubernetes resources which can be crafted to produce specific policies based upon many factors. These include:

• Namespaces: Container Admission Policies can be Namespace specific which enables flexibility and granularity depending on the Security controls required in each segment.
• Image Path: Policies can be written to only permit images located in specific registries to be deployed. This is necessary if Security governance mandates images can only be deployed from certain registries.
• Image Scan Status: The Admission Controller is able to retrieve the Image scan result from Calico Cloud and utilize them in policy. Policies can be created to permit or deny requests according to the image scan result.
• Image Last Scan: The date of the most recent scan can be used in policy to determine whether an image can be deployed. For example, a policy can be written to only permit images which have been scanned within a recent time period.

If we piece these components together then we can define the workflow as depicted above.

1. A user or automated CI/CID workflow requests to deploy a resource to the Kubernetes API.
2. The Kubernetes API makes a request to the Admission Controller asking whether the resource can be deployed. The image the resource is based upon is provided.
3. The Admission Controller fetches the image scan information from Calico Cloud which returns the Image scan result and date of the last scan.
4. Container Admission policies are evaluated by the Admission Controller to arrive at either an Allow or Reject action. Images with no Scan results sent to Calico Cloud return an Image status of Unknown which can be rejected if desired.
5. There are two actions depending on the Container Admission policy verdict:
   • Image scans resulting in a Reject action from step 4 will return a failure message back to the client indicating the deployment failed due to a Reject action in the Container Admission policy.
   • When an image matches an Allow action in a Container Admission policy rule the Kubernetes API is permitted to carry out the resource deployment/update.

Let’s use a real example to demonstrate the Admission Controller in use.

Calico Cloud in Action

In this previous blog post, we had used the image scanner to determine a number of known vulnerabilities which resulted in a Fail scan result. Now that the scan result is shared with Calico Cloud the Admission Controller is able to arrive at a decision on whether to allow or reject image deployments based upon the result.

Installation of the Admission Controller is detailed in Calico Cloud docs. At a high level the Admission Controller can be enabled per Namespace by use of labels. The ValidatingWebhookConfiguration resource shows the use of a namespaceSelector in order to filter which Namespace(s) require Admission Control:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: "image-assurance.www.tigera.io"
webhooks:
  - name: "image-assurance.www.tigera.io"
    namespaceSelector:
      matchExpressions:
        - key: "tigera-admission-controller"
          operator: "In"
          values:
            - "enforcing"
    rules:
      - apiGroups:   ["*"]
        apiVersions: ["*"]
        operations:  ["CREATE", "UPDATE"]
        resources:   ["pods", "replicasets", "deployments", "statefulsets", "daemonsets", "jobs", "cronjobs"]
        scope:       "Namespaced"
    clientConfig:
      service:
        namespace: "tigera-image-assurance"
        name: "tigera-image-assurance-admission-controller-service"
        path: "/admission-review"
        port: 8089
      caBundle: "**omitted**"
    admissionReviewVersions: ["v1", "v1beta1"]
    sideEffects: None
    timeoutSeconds: 5

Thus, any Namespaces with label tigera-admission-controller = enforcing will be subject to Calico’s Admission Control.

Next, once a Namespace is subject to the Admission Controller, Container Admission policies are required in order to determine the action, i.e. whether we allow or reject the deployment of the image. In our example we will allow the deployment of images only from the registry and repo tigeralabs.azurecr.io/chris-demo together with a scan result of Pass. The Container Admission policy we’ll use is as follows:

apiVersion: containersecurity.www.tigera.io/v1beta1
kind: ContainerAdmissionPolicy
metadata:
  name: reject-failed
spec:
  selector: all()
  namespaceSelector: all()
  order: 10
  rules:
  - action: 'Reject'
    imagePath:
      operator: IsNoneOf
      values:
      - '^tigeralabs.azurecr.io/chris-demo/.*'
  - action: Allow
    imageScanStatus:
      operator: IsOneOf
      values:
      - Pass
  - action: Reject

The namespaceSelector field enables us to use different policies per Namespace if we so wish. We can see there are 3 rules:

1. Reject any images with an image path that does not start with tigeralabs.azurecr.io/chris-demo/
2. Allow images with an image scan result of Pass. It is important to note the scan result of the image must be shared and uploaded to Calico Cloud.
3. Reject any other images.

The rules are evaluated sequentially and upon first match the configured action is taken. The configured Container Admission policy should meet our objective. Let’s test this. We have configured the Admission Controller ahead of time to apply only to Namespaces with the label tigera-admission-controller = enforcing. Next, we’ll create a Namespace, demo, and apply the required label:

❯ kubectl create ns demo
namespace/demo created

❯ kubectl label ns demo tigera-admission-controller=enforcing
namespace/demo labeled

Let’s attempt to deploy a pod using a forbidden registry:

❯ kubectl run -n demo busybox --image busybox:latest
Error from server (Action 'Reject' enforced by ContainerPolicy reject-failed rule index 0): admission webhook "image-assurance.www.tigera.io" denied the request: Action 'Reject' enforced by ContainerPolicy reject-failed rule index 0

We can see the Admission Controller denied this request due to the ContainerPolicy rule index 0 which corresponds to the registry/repo rule we configured.

Now we’ll attempt to deploy the image we previously scanned:

❯ kubectl run -n demo alpine --image tigeralabs.azurecr.io/chris-demo/alpine:3.10.1 --command -- sleep 300
Error from server (Action 'Reject' enforced by ContainerPolicy reject-failed rule index 2): admission webhook "image-assurance.www.tigera.io" denied the request: Action 'Reject' enforced by ContainerPolicy reject-failed rule index 2

The Admission Controller denied this request due to the ContainerPolicy rule index 2 which corresponds to the third rule which is our implicit deny rule. Therefore, we know the image did not match the first rule and thus originates from an allowed registry. The request also did not match the second rule which means the image scan result was not Pass. This makes sense because the images scan result is currently Fail. Therefore, as highlighted in the first blog post (mentioned at the beginning of this section), we might then want to look at the vulnerabilities which are causing the image scan to fail and decide whether the vulnerabilities are relevant to our application.

For our demonstration we’ll make exceptions for these high severity CVEs which changes the image scan result to Pass:

Now let’s retry deploying our image:

❯ kubectl run -n demo alpine --image tigeralabs.azurecr.io/chris-demo/alpine:3.10.1 --command -- sleep 300
pod/alpine created

Our image was now able to be deployed because the image originated from an approved registry and the image scan result is Pass.

Conclusion

Coupling Admission Control with Image scanning is an effective way to introduce build and deploy time security for container deployments. Admission Control needs to be flexible in order to meet the specific requirements for each application, service and/or environment. Calico’s Container Admission policies enable teams to create unique sets of policies which can be applied across various namespaces. Image scanning is essential for detecting known vulnerabilities in container images but equally important is the ability to detect unknown or zero day threats. Calico’s container threat detection provides runtime security by analyzing the behavior of containers and flagging any suspicious activity which could be indicative of a threat or compromise.

To try Vulnerability Assessment on Calico Cloud, get started with a free trial.