Using Calico Enterprise Flow Logs to Improve Visibility, Troubleshooting, and Collaboration
Watch this short video to see how you can benefit from better using Calico Enterprise Flow Logs.
What problems are we solving?
Distributed applications leverage multiple cross-connected services working together to deliver end-user functionality. These services are often maintained and updated by different teams, requiring Networking, Dev, DevOps, Platform, and Security teams to work in unison. There are many moving parts, as every aspect of the system can be modified by using:
kubectl apply -f
These factors make distributed applications very hard to troubleshoot. Traditional network troubleshooting tools simply do not work because of their lack of Kubernetes context, and their scalability limitations.
Downtime is very expensive and distributed application networking issues are difficult to troubleshoot. Platform, Networking and Security teams need an interface that can quickly and easily pinpoint the source of connectivity issues and facilitate the troubleshooting process. Security teams need an interface that can predict the behavior of microsegmentation policies. NOC and SOC teams need to be able to capture Kubernetes enriched logs in their logging and monitoring systems.
How does it work?
Calico Enterprise uses an Elasticsearch operator to deploy an Elasticsearch cluster and a Kibana instance. The Elasticsearch cluster is used to store flow logs according to specified retention settings to ensure the cluster does not run out of disk space. Elasticsearch and Kibana are integrated and managed as part of the Calico Enterprise lifecycle.
Architecturally, the logs are generated by the Calico data path component (Felix) into a file. The logs are pulled and transformed by a fluentd daemonset, and sent to Elasticsearch. Aggregation is handled by Felix. Filtering is handled by fluentd. Retention/recycling is handled by Elasticsearch.
Flow logs provide a rich source of information about every network connection in your Kubernetes cluster. In addition to NetFlow information, flow logs capture source/destination pods, namespaces, labels and policies.
The diagram, below, shows how flow logs are displayed using the Calico Enterprise Flow Visualize tool. It enables you to quickly drill down into a source-destination communication, and identify the root cause of a problem. Associated policies and allow/deny statistics simplify the troubleshooting process.
Why does this matter?
- Calico Enterprise flow logs enable you to trace the specific source/destination pod for any connection. By providing Kubernetes context (pod, labels, policy), we make it much easier for you to troubleshoot a given problem. The result is faster time-to-resolution, and less downtime.
- Network log retention is a requirement in most regulatory compliance mandates, and a necessity if you plan to conduct forensic analysis.
- Log retention can be costly in terms of data storage. Calico Enterprise provides tunable aggregation out of the box, which can help reduce storage costs by up to 90%.
How do I enable Flow Logs?
Flow logs are enabled by default. To use the flow logs, refer to the following interfaces:
- Flow Visualizer tool for connectivity and policy troubleshooting
- Kibana flow log interface for working with raw flow logs
- Kibana flow dashboard to use pre-build dashboards and create new ones
Need help configuring and tuning this feature? Please contact your Tigera Customer Success Engineer.
New Release! Calico Enterprise 2.7 is now available!
We are excited to announce the general availability of Calico Enterprise 2.7. We have added many new exciting capabilities that help platform engineers accelerate their deployments as well as improve their security posture. These include:
- Manage network security across multiple Kubernetes clusters
- Enforce a common set of security controls across multiple clusters
- Detect and alert on unauthorized changes and other attack vectors
- Enable self-service troubleshooting for end users
- Detect and prevent malicious data exfiltration
- Enable Fortinet FortiGate firewalls to enforce network security policies into and out of the Kubernetes cluster, as well as traffic between pods within the cluster
For a more detailed description of these capabilities, please read this blog.