Tigera serves the networking and policy enforcement needs of more than 150,000 Kubernetes clusters across the globe and supports two product lines: open source Calico, and Calico Enterprise. Our development team is constantly running smoke, system, unit, and functional verification tests, as well as all our E2Es for these products. Our CI pipelines form an extremely important aspect of the overall IT infrastructure and enable us to test our products and catch bugs before release.
We eventually reached a point where we needed to adopt a complete continuous integration and delivery architecture to maintain our development velocity, from code push to Kubernetes. We decided to adopt the hosted CI solution from Semaphore as an integral part of our workflow. Our test rigs on Semaphore ensure that the product is tested on three Kubernetes versions on seven different platforms, including Kubeadm, GKE, EKS, AKS, OpenShift, Rancher, and Kops. As a result, a typical pipeline can have up to 100 jobs distributed over various stages. We also have different pipelines to test our code and build Docker images for it.
Here’s a run for one of the components of our open source offering: All the stages shown can be translated into a simple pipeline file which you can generate using the Semaphore console.
Why did we choose Semaphore?
Traditionally, we used Jenkins for our CI needs and it served us well. But as we started to grow and expand our realm of testing to different versions of the product with different configuration options across various Kubernetes platforms we quickly realized that Jenkins might not be a good fit for us. Adopting Semaphore for our continuous integration needs enables us to focus on debugging test case failures and making overall product improvements rather than dealing with infrastructure maintenance issues. Semaphore is a good fit for us because of the native Golang support. We are a complete Golang shop! Semaphore has other awesome features which we totally love:
1. Easy onboarding: You can get a new project set-up on Semaphore with just a few clicks. If you’re not a fan of writing YAML files, it has a really cool, feature-rich workflow builder that lets you customize the pipeline on their console. The best part is that it renders the pipeline YAML for your reference.
2. Maintenance Free: One of the perks of using a hosted CI offering is that you don’t need to maintain in-house infrastructure for your ever-growing needs. If you feel you need more cores for your jobs to run, all you need to do is bump up quotas and it’s DONE!
3. Secrets Management: It is important to ensure that the secrets needed by your CI jobs do not live in the version control system. Semaphore treats secrets as first class citizens and you can store encrypted files, tokens, etc. in those secrets. Encrypted secrets can then be leveraged across multiple pipelines, jobs and stages.
4. Artifact Management: Often we need to pass artifacts from one stage to another. This is very easy with Semaphore. It enables us to share binaries and scripts, for example, which were modified in one stage with the job from another stage. And at the end of each workflow, we can use the artifact handling to push test results and diagnostic files for later review by team members.
5. Pipeline Dashboards: If you have close to 200 pipelines running each day with multiple stages, each with multiple jobs, digging out a particular pipeline can be a pain. We have a ton of pipelines running for different branches, pull requests and tags. Semaphore Dashboards helps us organize those pipelines so that we can find and view them easily. It also gives us a bird’s eye view of all our jobs.
For example, the CI pipeline for one of our internal dev tools looks something like this:
This is later deployed using a GitOps CD approach.
6. CLI support: Semaphore has an excellent CLI utility called sem CLI, which you can use to manage everything from projects to pipelines to secrets and dashboards. It’s extremely handy and provides a rich interface to all the features.
7. Notifications: The Semaphore console enables you to easily set notifications for different pipelines for different branches. You can have the notifications for a specific project delivered to a particular team’s Slack channel or consolidate all CI notifications in a single channel. Whatever works best for you. And yes, this too can be managed using the CLI.
8. Test case reporting: Our E2E runs trigger a number of tests and therefore digging into failures was always a little tricky with Jenkins. However, Semaphore has made our life easier. We can now look at all the failures consolidated for a particular run and dig deep into them by pushing relevant artifacts at the end of the job.
9. Pipeline scheduling: We use this feature heavily to trigger pipelines on a schedule, especially during non-work hours. Expecting runs to be green, every morning 🙂
10. Customer support: Much of our success using Semaphore can be attributed to their excellent customer support team. They are very responsive to issue escalations and also work to understand our use cases to improve existing features and deliver new ones.
Some final thoughts…
Experience has shown us that moving to a hosted CI service was the right decision. It’s helped us to improve developer productivity across the organization, and ensures that we are able to continually build and test the product with zero infrastructure maintenance overhead. We highly recommend using Semaphore for its excellent integrations and native support for various languages, frameworks, and tools like Docker and Kubernetes.
Want to learn more about Calico Enterprise? Check out these resources.
Free Online Training
Access Live and On-Demand Kubernetes Training
Calico Enterprise – Free Trial
Network Security, Monitoring, and Troubleshooting
for Microservices Running on Kubernetes