Securing the Modern Enterprise: Unified Microsegmentation and Observability with Calico

In the ever-evolving landscape of enterprise networks, the traditional approach of relying on a fortified perimeter to secure internal assets faces significant challenges. The dichotomy of a trusted internal network and an untrusted external environment, enforced by perimeter defenses, has been a longstanding strategy. However, the contemporary enterprise data center, characterized by a hybrid cloud model that incorporates on-premises infrastructure, cloud resources, and diverse cloud-native applications, presents a more complex security scenario.

Traditionally, organizations leaned on perimeter networks to establish a secure buffer between their trusted internal networks and the external internet. This security architecture employed measures like firewalls and intrusion detection systems to protect internal resources, control access, host public-facing services, and enforce authentication and authorization processes. While effective in its time, the evolving digital landscape has rendered traditional perimeter networks inadequate in addressing the dynamic nature of modern security threats.

Although perimeter networks still play a role in shielding internal endpoints against external threats, contemporary security architectures, such as Zero Trust Network Access (ZTNA), dismiss the concept of trusted and untrusted networks. In response to the dissolution of traditional network boundaries, microsegmentation emerges as a highly effective security measure. According to Gartner‘s “Priority Matrix for Workload and Network Security,” microsegmentation is a rapidly maturing security technology that takes less than two years to adopt and offers significant benefits.

Source: Hype Cycle for Workload and Network Security, 2023

This blog post delves into the intricate landscape of microsegmentation, exploring its principles, discussing the challenges associated with implementing an effective microsegmentation strategy, and providing a brief overview of Calico microsegmentation capabilities.

What is Microsegmentation?

Microsegmentation is a network security technique that involves dividing a network into smaller, isolated segments to enhance security by relocating the firewall to the host network interface. The core principle is to establish detailed security policies for individual workloads or applications, restricting communication based on specific criteria to thwart the lateral movement of attackers. This approach empowers security engineers to partition network traffic without extensive re-architecting, providing robust resistance against cyberattacks. Microsegmentation simplifies policy management by replacing address-based rules with an efficient identity-based policy system. By curtailing both north-south and east-west traffic, it seamlessly aligns with a zero-trust security strategy, significantly fortifying defense against cyber threats and strengthening overall network security posture.

Why Microsegmentation Matters

In today’s ever-changing cybersecurity landscape, where hybrid cloud setups, a surge in IoT devices, and always-on connectivity are the norm, microsegmentation stands out as a critical security measure. The convergence of on-premises infrastructure, diverse cloud platforms, and an expanding array of IoT devices introduces unparalleled complexity, challenging conventional security paradigms like network firewalls. While hybrid cloud models offer unprecedented flexibility, they also present intricate challenges in managing and securing data across diverse environments. The proliferation of IoT devices compounds these challenges, creating numerous endpoints vulnerable to cyber threats and spotlighting the shortcomings of conventional security measures. Microsegmentation is a potent solution to address these key business challenges:

• Enhancing Security Posture: Traditional network security approaches often rely on perimeter defenses such as firewalls, which may not provide sufficient protection against internal threats or the lateral movement of attackers, particularly in the face of advanced persistent threats (APTs). Microsegmentation introduces an extra layer of security by implementing granular policies at the level of individual workloads or applications, thereby enhancing overall defense mechanisms against sophisticated cyber threats.
• Supporting Compliance Requirements: Many industries and regulatory bodies require organizations to implement robust security measures to protect sensitive data and ensure compliance. Microsegmentation provides a framework for implementing and demonstrating compliance with these requirements by restricting access to sensitive systems and data only to authorized users and applications and ensure compliance with regulatory requirements such as GDPR, HIPAA, and PCI DSS.
• Facilitating Zero Trust Security: Microsegmentation aligns seamlessly with the principles of Zero Trust security, which operates under the assumption that threats can arise from both external and internal sources. By implementing stringent flow control rules and verification mechanisms irrespective of the user’s location or network segment, microsegmentation aids organizations in embracing a zero trust security model. This proactive approach significantly diminishes the risk of insider threats and curtails the lateral movement capabilities of potential attackers.
• Enhancing Operational Effectiveness: Identity-aware microsegmentation enhances operational efficiency by automatically assigning identities to workloads within the network, allowing new workloads to inherit pre-configured security policies based on their assigned identity. This automation streamlines deployment processes, reduces administrative overhead, and ensures consistent policy enforcement across all workloads, regardless of their location or movement within the network. By automating security policy enforcement based on workload identity, identity-aware microsegmentation optimizes network scalability and adaptability, enabling organizations to maintain robust security posture while effectively managing dynamic IT environments.
• Enhancing Auditing and Observability: Microsegmentation simplifies auditing processes within network infrastructures by dividing the network into smaller, manageable segments, enabling security engineers to apply specific security policies and access controls to each segment, facilitating easier monitoring of traffic flows and enhancing observability. This approach allows security engineers to quickly track and analyze network activity, identifying anomalies or unauthorized access attempts to ensure compliance with regulatory requirements.
• Protecting Diverse Workloads: A robust microsegmentation solution should support a diverse array of workloads, including physical servers, virtual machines (VMs), and containerized applications, regardless of their deployment location, whether it be in the cloud or on-premises.
• Enabling Secure DevOps Practices: In modern DevOps environments where applications are frequently deployed and updated, microsegmentation allows organizations to implement security policies that are dynamic and scalable. This enables secure communication between different components of distributed applications while maintaining isolation and minimizing security risks.

Microsegmentation Deployment Challenges

According to Gartner, “Microsegmentation is maturing quickly, but remains difficult to deploy at scale.” This section provides a brief overview of some of the challenges associated with implementing a microsegmentation solution.

• Implementation Framework: Depending on the size and extent of a datacenter’s footprint, the deployment of microsegmentation may demand a substantial initial investment and continuous maintenance efforts to safeguard upcoming workloads. A phased and progressive approach based on asset protection priority is recommended, constraining the implementation scope and advancing systematically. Additionally, a dynamic policy engine is essential for the effective implementation and sustainability of microsegmentation in the context of modern and dynamic workloads.
• Network Discovery and Observability: A significant obstacle in microsegmentation lies in identifying application dependencies and the necessary traffic connectivity details for each application. These details are crucial for crafting security policies. Manual determination of these specifics requires a nuanced understanding of communication patterns within the datacenter, a knowledge gap often prevalent in most organizations. Complicating matters further, many applications connect to ever-changing IP addresses in SaaS and other services, underscoring the critical need for up-to-date destination DNS information to implement domain-name-based security policies. The absence of a robust observability solution can render this process time-consuming, error-prone, and impactful.
• Policy Impact Preview: The potential impact of microsegmentation, if not implemented with due care and consideration, is significant. This challenge may trigger internal resistance, posing a threat to the entire project. To secure support from all internal stakeholders for a microsegmentation project, a robust solution is indispensable. This solution should seamlessly implement security policies, ensuring a non-disruptive process for live and production workloads. Additionally, it should provide sufficient visibility to swiftly validate connectivity information and smoothly transition into policy enforcement mode.
• Stakeholders engagement: As microsegmentation emerges as a powerful tool for mitigating risks in application environments, practical challenges come to the forefront during implementation. These challenges encompass a lack of understanding of existing application connectivity requirements, an abundance of policies requiring lifecycle management, inconsistent controls across dynamic workloads, and others. To address these challenges effectively, it’s imperative to identify and involve decision-makers and stakeholders in the deployment process. Emphasizing the human element, collaborative efforts among stakeholders are an absolute prerequisite for the seamless and successful deployment of microsegmentation.

Type of Microsegmentation Supported by Calico

Designed specifically for cloud-native workloads, Calico provides a robust, dynamic, and high-performance policy engine that seamlessly scales to tens of thousands of nodes and accommodates hundreds of thousands of application containers. What distinguishes Calico is its adoption of a simple, declarative, and user-friendly policy language, complemented by an intuitive policy user interface. This simplicity not only eases the creation of security policies but also enhances troubleshooting through the integration of Calico flow logs and observability capabilities, ensuring a seamless and efficient operational experience.

In the realm of microsegmentation, Calico employs two constructs—workload endpoints and host endpoints—to secure three categories of network endpoints: Kubernetes Pods, Kubernetes Cluster Nodes, and Non-Cluster Nodes.

• Kubernetes Pods: These are isolated and encapsulated units designed to host one or more containers. Kubernetes Pods enable efficient deployment and scaling of applications within a Kubernetes cluster. Calico employs a node agent, known as calico-node, which operates as a Kubernetes host networked pod to protect Kubernetes Pods. Kubernetes Pods are represented as workload endpoints in the Calico data model.
• Kubernetes Cluster Nodes: These individual servers collectively form the infrastructure within a Kubernetes cluster, serving as the foundation to run containerized workloads. In addition to Kubernetes Pods, calico-node provides built-in and native protection for Kubernetes Cluster Nodes. Cluster Nodes are represented as host endpoints in the Calico data model.
• Non-Cluster Nodes: Representing external servers or devices beyond the Kubernetes cluster, Calico’s microsegmentation extends its protective capabilities to secure bare-metal and virtual nodes, regardless of their deployment in the cloud or on-premises. To safeguard Non-Cluster Nodes, Calico provides an agent that needs to be deployed on these nodes, ensuring comprehensive protection across diverse network environments. Non-Cluster Nodes are also represented as host endpoints in the Calico data model.

Calico employs a holistic and unified approach to protect heterogeneous workloads. Calico policy engine utilizes a single and lightweight user-space agent functioning as a control plane component. It leverages existing firewall systems such as iptables, eBPF, Windows HNS container networking firewall, and others to implement security policies. The following section delves deeper into Calico’s security policy capabilities, illustrating how it empowers organizations to implement always-on protection for various types of workloads, whether in the cloud or on-premises.

Dynamic and policy-based microsegmentation

Cloud-native ecosystems are widely recognized for their dynamic and elastic features, empowering seamless workload scaling, efficient resource provisioning, and continuous application evolution. While these features offer significant benefits in terms of computation, application deployment, and orchestration, they present considerable challenges for conventional firewall systems. The need for security measures that are as dynamic and adaptive as the cloud-native workloads themselves becomes apparent.

In contrast to traditional models reliant on basic perimeter defenses, Calico offers a more advanced and effective approach to network security through policy-based segmentation. This strategy revolves around the creation, implementation, and enforcement of detailed, context-aware security policies designed to govern communication between diverse and dynamic workloads. By adopting policy-based segmentation, organizations can establish a simple and yet powerful method to secure their workloads within the ever-changing context of cloud-native environments.

Identity-Aware Microsegementation

Labels serve as the primary tool for grouping and organizing resources in the cloud-native ecosystem. Calico’s identity-aware microsegmentation leverages labels to identify network endpoints based on configured attributes such as operating system, datacenter, zone, department, app, and others, providing an easily describable and identifiable workload mechanism. These labels facilitate the development of flexible yet granular security policies for each workload. In the dynamic cloud-native security environment, Calico’s emphasis on labels is crucial for effective organization, identification, and the creation of precise security policies, ensuring continuous protection amid the ever-changing cloud environments. To illustrate, consider a scenario where network security policies are crafted based on workload identity, allowing, for example, all workloads in the “zone: trusted” and “datacenter: north-america” to establish egress connections to “app: active-directory” and  “zone: trusted” using TCP port 636, reinforcing the adaptability and effectiveness of Calico’s identity-aware microsegmentation.

Unified Security for Diverse Workloads

Calico utilizes the Kubernetes datastore in conjunction with the Kubernetes control plane to manage security policies for Kubernetes Pods, cluster nodes, and non-cluster nodes. The Kubernetes datastore serves as the authoritative source for the Calico policy engine, utilizing stored policy information to synchronize the policy agent dataplane with the desired state. The Calico policy engine empowers security engineers with a unified control plane, management interface, and policy language, simplifying microsegmentation across diverse workloads.

Scalability and Performance

To avoid overloading the Kubernetes control plane with excessive queries from Calico policy agents and ensure efficient scalability for a large number of nodes, Calico employs Typha—a cluster of highly available reverse proxy pods. Typha utilizes mTLS to keep the policy agents updated with the latest policies and policy changes. Refer to the following diagram for a visual representation.

As previously mentioned, Calico’s security policy model relies on a lightweight user-space control plane agent. This agent leverages the node’s built-in firewall systems, such as iptables for Linux and the Windows HNS container networking firewall, to enforce security policies. The extensive usage of the Calico policy engine in large-scale environments, encompassing thousands of nodes, tens of thousands of namespaces, and numerous security policies, has led to optimizations for consistent performance in such expansive settings. Calico is not only optimized to run in environments with massive cluster and node footprints but also meets the performance demands of applications requiring minimal latency. In the latter case, certain application and operating system tuning, along with Calico security policy best practices become a necessity.

Single Pane of Glass Observability

Flow logs

To design and implement effective microsegmentation policies, it’s crucial to have comprehensive visibility into network flows and identify application dependencies. Once the security policies are in place, maintaining a clear view of network traffic becomes essential to ensure legitimate application communications while blocking unauthorized access. Furthermore, the availability of real-time and historical flow data is a requirement for demonstrating adherence to regulatory standards. The Calico policy agent collects and provides enriched flow information across various endpoint types it protects. In addition to conventional 5-tuple data, Calico flow logs offer detailed insights into enforced security policies, endpoint labels, process details, TCP statistics, and more. Tailored specifically for Kubernetes workloads, Calico flow logs furnish essential metadata such as namespace and deployment information for containerized workloads. Leveraging flow logs, Calico Service Graph and custom Kibana dashboards provide a comprehensive view of network traffic flows, empowering security engineers to seamlessly comprehend, manage, secure, and troubleshoot the connectivity of various workload types. For further information on Calico flow logs, visit this blog post.

DNS Logs

The Calico DNS security policy enables security engineers to authorize egress access to destination domain names, even when they lack control over the associated IP addresses. This is especially crucial for scenarios where IP addresses frequently change, necessitating a security policy based on DNS names. Implementing DNS-based security policies requires a deep understanding of the specific domain names that applications need to communicate with. While engineers typically grasp an application’s ingress connectivity requirements well, they often lack awareness of its egress connectivity needs, particularly in cloud environments. In such dynamic settings, applications may require connectivity to numerous destination domain names via proxy services.

Calico DNS logs provide comprehensive metadata on name resolution activities, including the source of the request, the DNS server responding to the request, latency information, and more. These logs serve as invaluable sources of information for observability, troubleshooting, and policy development, enabling security engineers to gain insights into the endpoints DNS behavior and make informed decisions.

Shift-Left Security and DevOps

Calico provides a suite of advanced policy tooling, including a policy UI for crafting sophisticated policies, Policy Recommendation for automated policy suggestion and creation, Staged Policies for auditing policy impacts without traffic interference, and RBAC and admission control for precise security policy management permissions. These tools streamline policy creation, enhance observation without affecting traffic, and enable effective control over access and traffic flow, empowering security engineers to efficiently manage policies within a Calico environment. By integrating security measures earlier in the development lifecycle and aligning them with the rapid iterative processes of DevOps, Calico’s policy engine capabilities facilitate a proactive shift-left security approach. Developers can define and enforce fine-grained network policies at the application level, specify security requirements directly within their code or configuration files, and leverage automation features like Policy Recommendation and Staged Policies to seamlessly test and iterate on security policies within their development environments. This collaborative and efficient approach ultimately leads to faster and more secure application delivery.

Software Defined Security

To safeguard both cluster and non-cluster nodes, Calico relies on the HostEndpoint resource to secure incoming and outgoing traffic. The HostEndpoint acts as a comprehensive profile, incorporating various configuration parameters like node labels, node name, IP addresses configured on the node, and more additional settings. Notably, this profile operates independently of the underlying hardware, guaranteeing that as long as the node maintains its designated name and expected IPs, its traffic stays protected. This strategy not only streamlines node upgrades, encompassing complete hardware replacements or migrations to different environments but also enhances the network’s resilience and security posture.

For nodes requiring connectivity to various networks with distinct connectivity requirements and security profiles, Calico offers the flexibility to create multiple HostEndpoint resources. These resources allow for the application of different sets of security policies tailored to each node’s network interface, ensuring compliance with specific security requirements. By leveraging multiple HostEndpoint resources, administrators can effectively manage diverse network configurations while maintaining robust security measures across the board. This adaptable approach further strengthens Calico’s ability to meet the evolving needs of complex network environments.

Although administrators are responsible for creating and managing HostEndpoint instances, Calico offers an automated solution called auto HostEndpoint specifically designed for cluster nodes. This feature is fully managed and maintained by Calico, requiring minimal configurations to be activated. As cluster nodes are dynamically added or removed from the cluster, Calico’s security policy agent seamlessly handles the creation and deletion of HostEndpoints. This transparent process ensures continuous protection without requiring direct administrator involvement. Below is an example of HostEndpoint resource.

apiVersion: projectcalico.org/v3
kind: HostEndpoint
metadata:
  labels:
    hep.non-cluster-node: "true"
    hep.app: kafka
    hep.component: zookeeper
    hep.zone: trusted
    hep.dc: westus
    hep.env: production
    hep.network: production
  name: node1.tigera.io
spec:
  expectedIPs:
  - 10.0.1.1
  - 10.0.1.100
  node: node1.tigera.io
  profiles:
  - projectcalico-default-allow

Summary

Traditional perimeter-based security models are proving inadequate in today’s rapidly changing digital landscape and dynamic cloud environments. To address these challenges effectively, microsegmentation has emerged as a powerful security measure. This article explores the fundamental principles of microsegmentation, its crucial role in modern cybersecurity, and its diverse application in enhancing security postures, meeting compliance requirements, and enabling zero-trust security approaches. Additionally, it discusses the challenges associated with implementing microsegmentation solutions, emphasizing the need for a phased deployment approach, robust observability tools, the ability to preview policy impacts, and engaging stakeholders proactively.

The discussion extends to Calico’s approach to microsegmentation, highlighting its flexibility in supporting various workloads—be it physical servers, virtual machines, or containerized applications—while also highlighting its scalability and performance capabilities. Moreover, Calico’s observability features, including flow logs and DNS logs, furnish invaluable insights into network activities, enhancing threat detection and facilitating swift incident responses. The integration with DevOps practices further empowers organizations to embed security early in the development lifecycle, fostering a proactive “shift-left” security approach. Calico offers a comprehensive solution that adeptly tackles the challenges associated with microsegmentation deployment, equipping organizations with the requisite tools to enhance their security postures amid today’s intricate and dynamic cybersecurity landscape.

Ready to try Calico network policies? Sign up for a free trial of Calico Cloud