How to integrate Calico Image Scanner with Argo CI/CD

In today’s fast-paced software development environment, developers often use common public libraries and modules to quickly build applications. However, this presents a significant challenge for DevOps teams who must ensure that these applications are safe to use. As organizations move towards dynamic models of software development that rely on Continuous Integration and Continuous Deployment, the responsibility for deploying secure applications has shifted from traditional security teams to development teams.

To address this challenge, I will provide general guidelines on how to integrate the Calico Image Scanning feature into a CI/CD pipeline, using Argo. This will help ensure that images are built safely and free from Common Vulnerabilities and Exposures (CVEs). In this blog post, we will use a Kubernetes validating webhook configuration to attach a Calico Cloud admission controller that can accept or reject certain actions on resources, such as the creation of pods. This will prevent the deployment of images that contain known CVEs, thus strengthening the overall security of your software development process.

Overall Architecture

The building blocks to use Argo as an example of this integration are below:

Relevant Calico configuration

Before even committing changes to our application, we must setup the Calico Admission Controller within our Calico Cloud setup, as detailed in our Docs page.

Then an “admissioncontrollerpolicy” resource must be created. In this example, we  will be preventing anything that has a CVSS score of “Failed” to be deployed in our environment (For other operations, please check the relevant resource doc page.

apiVersion: containersecurity.www.tigera.io/v1beta1
  kind: ContainerAdmissionPolicy
  metadata:
    ...
    name: reject-failed
    ...
  spec:
    namespaceSelector: all()
    order: 10
    rules:
    - action: Allow
      imageScanStatus:
        operator: IsOneOf
        values:
        - Pass
        - Warn
    - action: Reject
    selector: all()

Sequence of events

To test the above, this will be the sequence of events in our pipeline:

1. A webhook in the repo sends any push event done to an ingress controller.
2. This specific k8s ingress has a service whose endpoint is an “eventsource” resource in the Argo Events namespace.
3. The “eventsource” receives the event, and then triggers a “sensor” resource in Argo Events which creates a corresponding workflow.

{"level":"info","ts":1679935064.284753,"logger":"argo-events.eventsource","caller":"github/start.go:130","msg":"received a request, processing it...","eventSourceName":"github","eventSourceType":"github","eventName":"example","endpoint":"/push","port":"12000","http-method":"POST"}
{"level":"info","ts":1679935064.2853296,"logger":"argo-events.eventsource","caller":"github/start.go:164","msg":"dispatching event on route's data channel","eventSourceName":"github","eventSourceType":"github","eventName":"example","endpoint":"/push","port":"12000","http-method":"POST"}
{"level":"info","ts":1679935064.2854376,"logger":"argo-events.eventsource","caller":"github/start.go:166","msg":"request successfully processed","eventSourceName":"github","eventSourceType":"github","eventName":"example","endpoint":"/push","port":"12000","http-method":"POST"}
{"level":"info","ts":1679935064.2854986,"logger":"argo-events.eventsource","caller":"webhook/webhook.go:187","msg":"new event received, dispatching it...","eventSourceName":"github","eventSourceType":"github","eventName":"example"}
{"level":"info","ts":1679935064.2905464,"logger":"argo-events.eventsource","caller":"eventsources/eventing.go:558","msg":"Succeeded to publish an event","eventSourceName":"github","eventName":"example","eventSourceType":"github","eventID":"65313037333131362d346161322d343363622d613833322d303265656563316636333338"}

…

{"level":"info","ts":1679935064.3465035,"logger":"argo-events.sensor","caller":"sensors/listener.go:417","msg":"Successfully processed trigger 'github-workflow-trigger'","sensorName":"github","triggerName":"github-workflow-trigger","triggerType":"Kubernetes","triggeredBy":["test-dep"],"triggeredByEvents":["65313037333131362d346161322d343363622d613833322d303265656563316636333338"]}

4. This workflow will build an image with the required changes, and then a second workflow will create another pod with a binary (the Tigera CLI scanner) which will look for vulnerabilities in this new image. Those scan results will be sent to Calico Cloud, so we will have not just the option of seeing those, but Calico Cloud will report if any running container is using such image in case we did not implement our Admission Controller before they were deployed:

5. Finally, ArgoCD will try to deploy the required applications with the new  image based on what is defined in the repo it syncs from.
6. Beforehand, we applied an “admissioncontainerpolicy” resource that Calico manages. This webhook will prevent deploying any image that does not meet the criteria outlined in the previous section (“Relevant Calico configuration”). In our example, the applied  policy will prevent the image if there is any vulnerability which has a CVSS score above the threshold of what will be considered a “Warn” value. We can see that for a failing test case, we will have a relevant entry in the application controller:

% kubectl logs -n argocd argocd-application-controller-0  | grep Reject | tail -1
time="2023-03-29T09:47:20Z" level=info msg="Updating operation state. phase: Running -> Failed, message: 'one or more tasks are running' -> 'one or more objects failed to apply, reason: admission webhook \"image-assurance.www.tigera.io\" denied the request: Action 'Reject' enforced by ContainerPolicy reject-failed rule index 1'" application=argocd/java-app syncId=00014-WkFyP

And the relevant application will be shown as “Out of sync” in ArgoCD:

Conclusion

Cloud native applications often rely on open source software and libraries, which can pose a significant threat to the security of your Kubernetes environment. The widespread use of these components means that a single vulnerability can quickly spread throughout your system. Fortunately, Calico Cloud offers a solution by preventing dangerous images from being deployed at build time through close integration with your CI/CD pipeline.

If you’re interested in learning more about how Calico Cloud can help secure your Kubernetes environment, we invite you to watch our brief 3-minute video. This video will guide you through the necessary steps to integrate Calico Cloud into your CI/CD pipeline.

Ready to get started? Try a free trial.