Cilium and eBPF Networking Architecture Explained
Modern Kubernetes networking is increasingly implemented through programmable Linux kernel datapaths. Cilium uses extended Berkeley Packet Filter (eBPF) to deliver container networking, service load balancing, identity-aware security and deep network observability.
Instead of relying exclusively on traditional Linux networking mechanisms such as iptables, Cilium can attach verified eBPF programs to supported kernel hooks, allowing network decisions to occur close to where packets and socket operations are processed.
This guide explores the relationship between Kubernetes workloads, the Cilium control plane, eBPF programs and maps, the Linux kernel networking stack, Hubble observability and external network connectivity.
Explore Four Cilium Engineering Domains
Select an engineering domain to examine how Cilium connects Kubernetes networking, Linux kernel execution and operational visibility.
eBPF Packet Forwarding
Cilium attaches supported eBPF programs to Linux kernel hooks to implement packet processing, policy checks, forwarding and service connectivity. The precise path depends on the configured Cilium datapath.
The Three Layers of Cilium Networking
Kubernetes Workloads
Pods, services and namespaces define application endpoints and desired connectivity. Kubernetes resources provide the inputs used by Cilium to configure networking behaviour.
Cilium Control Plane
The Cilium agent manages endpoint state, security identities, service information and datapath configuration. The operator performs supported cluster-level management tasks.
Linux eBPF Datapath
Verified eBPF programs execute at supported kernel hooks, using maps and kernel helpers to make forwarding, service and security decisions.
From Kubernetes Pod to Network Destination
The exact processing sequence differs between same-node traffic, cross-node traffic, service traffic and external network flows. The following sequence is a simplified conceptual model.
Why Cilium Changes Kubernetes Networking
Cilium introduces a programmable networking model where supported forwarding and security functions can be implemented through eBPF rather than relying exclusively on conventional packet-processing mechanisms.
This provides opportunities for efficient service handling, workload identity-based security, detailed flow visibility and integration with external routing systems.
However, eBPF does not remove the need to understand the underlying Linux network stack. Engineers still need to investigate routing, MTU, conntrack state, kernel compatibility, endpoint identities, policy behaviour and packet drops.
Core engineering principle: Cilium's user-space components configure the desired networking behaviour, while supported eBPF programs enforce and implement datapath decisions inside the Linux kernel. Successful troubleshooting requires visibility into both layers.
Linux kernel hooks, eBPF programs and maps, Cilium agent, operator, endpoint identities and the Kubernetes CNI lifecycle.
Cilium and eBPF Networking Architecture
Explore the Linux kernel datapath, Kubernetes networking, eBPF service load balancing, Zero Trust policies, Hubble observability, BGP routing, Cluster Mesh and production troubleshooting through nine interconnected engineering sections.
Explore All Nine Engineering Sections
Select any architecture domain to jump directly to its editorial section.
Cilium and eBPF Architecture Fundamentals
Cilium is a cloud-native networking, security and observability platform that uses extended Berkeley Packet Filter (eBPF) to execute programmable networking functions inside the Linux kernel.
Its architecture connects Kubernetes configuration and workload identity to kernel-level packet processing. The Cilium agent interprets cluster state, manages endpoints and policy, and configures the eBPF datapath. The Linux kernel then executes verified programs at supported attachment points.
Understanding this separation between control-plane orchestration and kernel datapath execution is essential for engineering, securing and troubleshooting Cilium deployments.
How Kubernetes, Cilium and the Linux Kernel Work Together
A Cilium-managed Kubernetes node contains workload endpoints, user-space management components and kernel-resident eBPF programs and maps. The components collaborate, but they do not all execute in the packet path.
From eBPF Source Code to Kernel Execution
eBPF programs are commonly compiled from C or other supported languages into eBPF bytecode. A privileged loader submits the program to the kernel, where the verifier checks program safety and permitted execution behaviour before it can be loaded.
On supported platforms, just-in-time compilation can translate verified eBPF bytecode into native machine instructions. The resulting program executes when its configured kernel hook is triggered.
Important distinction: The verifier does not establish that a networking policy is operationally correct. It checks whether a program satisfies kernel verification constraints. Network behaviour must still be validated through policy testing, packet-path inspection and observability.
Where eBPF Programs Interact with Networking
| Attachment Point | Typical Use | Engineering Consideration |
|---|---|---|
| XDP | Early packet processing on supported network devices. | Runs before much of the conventional networking stack; feature and driver support matter. |
| TC | Ingress and egress packet processing. | Commonly used for endpoint forwarding, policy and service operations. |
| Socket / cgroup hooks | Socket-level service and connectivity decisions. | Can implement selected service translation and policy functions before packet forwarding. |
| Tracing hooks | Kernel event and diagnostic instrumentation. | Useful for observability; not every tracing hook is a forwarding hook. |
The exact attachment points used by Cilium depend on the configured datapath, enabled features, Linux kernel version and network-device capabilities. XDP, TC and socket hooks should not be treated as mandatory sequential processing stages for every packet.
How Programs Share Networking State
eBPF maps are kernel-managed data structures that allow supported programs and user-space components to exchange and maintain state. Cilium uses maps for functions such as endpoint identity lookup, service backend selection, connection tracking and policy-related decisions.
Map capacity, lookup performance, lifecycle management and state consistency are important operational considerations. Stale or exhausted map state can contribute to connectivity failures and should be investigated alongside routes and security policy.
Separating Node-Level and Cluster-Level Responsibilities
Cilium Agent
Typically deployed on each managed node. Coordinates endpoint state, identities, policy, service information and datapath configuration.
Cilium Operator
Performs supported cluster-level functions such as coordinating certain IPAM, identity and resource-management operations, depending on configuration.
Linux Kernel Datapath
Executes loaded eBPF programs and maintains relevant map state, allowing supported forwarding decisions to occur without a user-space round trip for every packet.
Control-plane versus data-plane principle: An existing kernel datapath may continue processing supported traffic during some user-space control-plane interruptions, but new endpoint provisioning, policy updates and other operations can be affected. Behaviour depends on the failure and configuration.
How a Pod Becomes a Cilium-Managed Endpoint
When Kubernetes schedules a pod, the container runtime invokes the configured CNI integration to establish networking. Cilium participates in provisioning the endpoint, configuring the relevant networking interfaces and preparing the datapath for the workload.
Troubleshooting pod connectivity therefore requires examining both Kubernetes lifecycle state and Cilium endpoint state. A running container is not sufficient evidence that all networking configuration and policy has converged.
Moving Beyond IP-Based Network Security
Cilium uses security identities derived from relevant workload labels and metadata. These identities allow policy to describe relationships between groups of workloads rather than relying exclusively on fixed IP addresses.
This is especially useful in Kubernetes environments where pods are frequently recreated and IP addresses change. Identity-based policy can remain aligned with workload intent as endpoint addresses evolve.
Policy semantics depend on the selected endpoints, direction, protocol, policy resources and configured enforcement mode. Identity-aware security complements, rather than automatically replaces, application authentication and authorisation.
What Engineers Should Verify Before Deployment
The Foundation of Cilium Networking
Cilium combines Kubernetes-aware control-plane management with programmable Linux kernel execution. The agent and operator coordinate supported networking state, while verified eBPF programs and maps implement selected forwarding, service and security functions.
The fundamental architecture can be summarised as: Kubernetes intent → Cilium control plane → eBPF programs and maps → Linux kernel execution → validated network behaviour.
Next section: eBPF Datapath and Packet Processing — a deeper investigation of TC, XDP, socket hooks, packet redirection, map lookups, service translation and the forwarding decisions made as traffic moves between Kubernetes workloads and external networks.
Cilium and eBPF Architecture Explorer
Explore how Cilium connects Kubernetes control-plane information with programmable networking inside the Linux kernel. Investigate the Cilium agent, operator, endpoint identities, eBPF programs, maps, CNI integration, and the forwarding path between Kubernetes workloads.
Kubernetes, Cilium and Linux Kernel Architecture
Select any component to inspect its role. Gold paths represent control-plane interactions; green paths represent packet forwarding.
Architecture Engineering Console
Architecture State Controls
The Cilium agent manages and reconciles datapath state. Existing kernel programs may continue forwarding traffic during a temporary agent interruption.
Cilium Control-Plane and Kernel Inventory
| Component | Execution | Responsibility | Status |
|---|
Packet Processing Investigation
eBPF Architecture Diagnostic Console
Engineering Interpretation
Production Engineering Notes
Cilium typically deploys an agent on each Kubernetes node. The agent consumes Kubernetes information, manages endpoints, programs relevant eBPF maps, and attaches datapath programs at supported kernel hooks. The Cilium operator handles selected cluster-wide functions, depending on configuration and enabled features.
eBPF execution occurs inside the Linux kernel, while program loading and orchestration occur in user space. Maps provide kernel-accessible state for networking functions such as identities, connection tracking, service translation, and policy decisions.
TC, XDP, and socket hooks are different execution points. XDP operates early on supported ingress interfaces, TC can process packets at traffic-control hooks, and socket-level programs can support selected service and connection operations. They are not necessarily sequential stages of every packet's journey.
Educational simulation. The topology represents conceptual responsibilities, not a complete kernel call graph. Real Cilium behavior depends on the installed version, kernel capabilities, routing mode, enabled features, and configured network policies.
eBPF Datapath and Packet Processing
The Cilium datapath uses Linux eBPF programs to implement selected networking, security and service-handling functions at supported kernel attachment points.
Rather than treating every packet as a sequence of traditional iptables rules, Cilium can use programmable kernel hooks and shared eBPF maps to make forwarding decisions based on endpoint identity, service state, connection tracking and network policy.
Understanding this datapath is essential for diagnosing packet drops, service failures, routing problems and unexpected policy behaviour in Kubernetes.
Following a Packet Through the Linux Kernel
A packet entering or leaving a Kubernetes workload may encounter eBPF programs attached to supported networking hooks. The exact processing path depends on the traffic type and the configured Cilium datapath.
Socket-level service handling, TC-based packet processing and XDP-based early packet processing represent different execution opportunities rather than a mandatory sequence for every packet.
TC, XDP and Socket-Level eBPF
eBPF programs can execute at different points in the Linux networking stack. Each hook provides a different context and set of supported actions.
Traffic Control (TC)
TC ingress and egress hooks can support packet processing, policy enforcement, forwarding and service handling on relevant network interfaces.
Express Data Path (XDP)
XDP provides early packet processing for supported network devices and drivers, enabling selected filtering and forwarding operations before normal stack processing.
Socket / cgroup Hooks
Supported socket hooks can influence connection and service handling before conventional packet forwarding occurs.
Important: TC, XDP and socket hooks are different execution contexts. A packet does not necessarily traverse all three. Engineers must identify the actual configured hook when investigating behaviour.
How Kernel Programs Make Forwarding Decisions
eBPF maps provide data structures that allow programs to retrieve endpoint, service, policy and connection state.
The datapath can use map lookups to select a service backend, determine an endpoint identity or evaluate relevant forwarding and security information.
Pod-to-Pod and Pod-to-Service Forwarding
Pod-to-pod traffic and pod-to-service traffic may follow different processing paths.
Direct pod connectivity generally requires endpoint routing and policy evaluation. Service traffic additionally requires a mechanism to select an appropriate backend.
The sequence above is a conceptual representation. In supported configurations, socket-level load balancing may occur before packet-level processing, and connection tracking can influence subsequent packets in a flow.
Forwarding Without Traditional Per-Packet User-Space Processing
eBPF programs can use supported kernel mechanisms to redirect packets, select interfaces or influence forwarding decisions without requiring every packet to be passed to a user-space networking application.
Depending on configuration, Cilium may use native routing, encapsulation or other supported datapath mechanisms to deliver packets between nodes and endpoints.
Performance principle: eBPF can reduce certain packet-processing overheads, but performance depends on workload characteristics, kernel capabilities, map access patterns, encapsulation, policy complexity and the selected forwarding mode.
Stateful Flow Handling in the eBPF Datapath
Connection tracking allows the datapath to maintain information about supported network flows.
This state can be used to support service translation, forwarding consistency and security enforcement. Engineers should distinguish Cilium-managed BPF connection tracking from other Linux connection-tracking mechanisms.
Why eBPF Datapath Packets May Be Dropped
A packet drop can result from network policy, missing routing state, unsupported forwarding conditions, invalid service state or other datapath constraints.
| Failure Category | Possible Cause | Investigation |
|---|---|---|
| Policy denial | Traffic not permitted by applicable policy. | Inspect endpoint identity and policy verdict. |
| Routing failure | Destination or forwarding information unavailable. | Inspect routes, endpoint state and datapath configuration. |
| Service failure | Backend selection or service state problem. | Inspect service and backend mappings. |
| Conntrack issue | State exhaustion, inconsistency or timeout. | Review BPF conntrack maps and relevant metrics. |
| MTU / encapsulation | Packet size or tunnel overhead issue. | Validate MTU, path MTU and forwarding mode. |
Inspecting the Cilium Datapath
Cilium provides diagnostic capabilities for examining endpoint state, service mappings, policy behaviour, BPF maps and datapath events.
Understanding the Programmable Kernel Datapath
The Cilium eBPF datapath provides a programmable networking architecture in which supported kernel programs and maps implement selected forwarding, security and service functions.
The essential processing model is: workload traffic → supported kernel hook → eBPF logic and state → forwarding or drop decision → destination.
Next section: Kubernetes Pod Networking and IPAM — endpoint addressing, CNI provisioning, pod routing, native versus overlay networking, VXLAN and Geneve, multi-node forwarding and production connectivity troubleshooting.
eBPF Packet Journey Simulator
Follow a packet through the Linux networking stack and investigate how Cilium uses eBPF programs, kernel hooks, maps, and forwarding decisions to implement high-performance Kubernetes networking.
Linux Kernel eBPF Datapath Architecture
The visualization separates early ingress, TC processing, map lookup, and forwarding. Socket-level load balancing is modeled as a separate execution perspective.
eBPF Packet Engineering Console
Kernel State Controls
eBPF Map and Hook Inspector
| Kernel Component | Function | Status |
|---|
Map categories are conceptual. Actual Cilium installations use multiple maps and program types whose names and availability depend on version and enabled features.
Packet Processing Pipeline
Kernel Packet Diagnostic Console
Engineering Interpretation
Production Engineering Notes
XDP, TC, and socket-level eBPF hooks operate at different points in the Linux networking stack. XDP can process ingress traffic before normal socket-buffer allocation on supported drivers and interfaces. TC programs process packets through traffic-control attachment points.
Cilium uses eBPF maps to maintain networking state, including connection tracking, policy decisions, service translation, and endpoint information. These operations depend on the configured datapath and supported kernel capabilities.
Socket-level service load balancing can translate destinations at connection-related hooks, avoiding some later packet-level translation work. It should not be modeled as another mandatory stage after XDP and TC for every packet.
Educational simulation only. The packet animation is a conceptual teaching model, not an exact Linux kernel call graph or a real-time trace from a running Cilium cluster.
Kubernetes Pod Networking and IPAM
Cilium connects Kubernetes workloads through a programmable datapath while managing endpoint addressing, network interfaces, forwarding state and policy integration.
Every Kubernetes pod requires a network identity and a path to other workloads. Cilium supports multiple IP address management (IPAM) approaches and both native-routed and encapsulated networking designs.
The choice of IPAM and forwarding mode determines how pod addresses are allocated, how nodes reach remote pod networks, which routes the underlying infrastructure must understand, and how engineers troubleshoot packet delivery.
How Cilium Connects Pods Across Kubernetes Nodes
The architecture below shows two Kubernetes nodes, each with its own pod network. Select a forwarding mode to compare how traffic can move between nodes.
Traffic to a remote pod is forwarded using routes that ultimately provide reachability to the destination pod address. The underlay or node routing architecture must support the relevant pod-prefix forwarding.
From Pod Creation to a Working Network Endpoint
Kubernetes delegates pod network setup to the container runtime and configured CNI implementation. Cilium's integration provisions endpoint networking and coordinates datapath state with the running Cilium agent.
Pod Network Environment
The runtime establishes the pod sandbox and its network namespace, creating the environment into which networking interfaces are configured.
IPAM and Interfaces
The selected IPAM implementation assigns or coordinates pod addresses. Cilium configures the appropriate interface and routing integration.
Identity and Datapath
Cilium manages the endpoint, determines its security identity, and prepares relevant eBPF policy and forwarding state.
Engineering distinction: A pod receiving an IP address does not prove that cross-node routing, service connectivity or security-policy state is correct. Validate each layer independently.
Understanding Cilium IPAM Models
Cilium supports several IPAM strategies. The available options and their operational requirements depend on the deployment platform, Cilium version and cloud integration.
| IPAM Approach | Architecture | Operational Consideration |
|---|---|---|
| Kubernetes PodCIDR | Uses pod CIDRs allocated to nodes by Kubernetes, with Cilium managing addresses from those ranges. | Requires appropriate Kubernetes node CIDR allocation and routing design. |
| Cluster-Pool IPAM | Cilium operator allocates per-node pod CIDR ranges from configured cluster pools. | Plan pool capacity, mask sizes and non-overlapping ranges. |
| Multi-Pool IPAM | Supports selecting addresses from multiple configured pools in supported deployments. | Requires explicit pool design and compatibility validation. |
| Cloud ENI / Cloud IPAM | Integrates with supported cloud networking address allocation, such as AWS ENI-based designs. | Cloud quotas, subnet capacity, permissions and node interface limits matter. |
For example, a cluster-pool design could reserve 10.0.0.0/16 for workload addressing and assign separate /24 pod ranges to individual nodes. These are illustrative planning values, not Cilium defaults.
When the Physical or Cloud Network Routes Pod Traffic
In native-routing mode, Cilium does not require an encapsulation overlay for ordinary cross-node pod forwarding. Remote pod addresses must instead be reachable through the routing architecture.
Routes may be established through the underlying network, node routing mechanisms or supported route-advertisement integrations. The exact design depends on the cluster topology.
Production risk: If the underlay cannot reach the destination pod network, an otherwise healthy Cilium endpoint may still experience cross-node connectivity failures.
VXLAN and Geneve Overlay Networking
Overlay networking transports pod traffic inside an outer packet addressed between reachable tunnel endpoints, typically Kubernetes nodes. This reduces the requirement for the underlay to maintain routes to every pod prefix.
Cilium supports VXLAN and Geneve in relevant tunnelling configurations. The selected mode adds encapsulation overhead, which must be considered when determining workload MTU.
| Property | Native Routing | Overlay Routing |
|---|---|---|
| Pod-prefix visibility | Underlying routing must provide pod reachability. | Underlay primarily requires tunnel endpoint reachability. |
| Encapsulation | No Cilium overlay encapsulation for the native path. | VXLAN or Geneve encapsulation. |
| MTU | Depends on underlying path. | Must account for tunnel overhead. |
| Operations | Investigate pod routes and underlay forwarding. | Investigate tunnel state, node reachability and MTU. |
| Tradeoff | Potentially simpler packet format, greater routing integration. | Simpler underlay requirements, additional encapsulation complexity. |
Tracing Pod A to Pod B
Consider a frontend pod on Node A communicating with a backend pod on Node B. The packet must pass the applicable policy and forwarding decisions and then traverse the inter-node network.
The actual order and location of policy checks, routing lookups and connection-tracking operations depend on the enabled datapath features. This sequence is intended as an engineering investigation model.
Why Pod Connectivity Breaks
| Symptom | Likely Investigation Area | Validation |
|---|---|---|
| Same-node works; cross-node fails | Underlay routing, tunnel reachability, firewall rules or MTU. | Compare node-to-node and pod-to-pod paths. |
| Pod cannot obtain an address | IPAM pool capacity, operator, cloud quotas or CNI lifecycle. | Inspect Cilium status and pod events. |
| Only large packets fail | MTU or path-MTU discovery. | Test packet sizes and inspect tunnel overhead. |
| Selected pods cannot communicate | Identity resolution or network policy. | Inspect endpoint identity and flow verdicts. |
| Connectivity fails after node replacement | Stale routes, tunnel mappings or endpoint state. | Compare current node addressing with datapath state. |
Validating Cilium IPAM and Forwarding
Begin with endpoint readiness and assigned addresses, then identify whether the failure is in the local datapath, cross-node network or destination endpoint.
Address Allocation and Forwarding Must Be Designed Together
Cilium pod networking depends on both correct endpoint provisioning and a reliable forwarding architecture. IPAM determines which addresses workloads receive; native routing or overlay tunnelling determines how packets reach remote nodes and pods.
The essential engineering model is pod lifecycle → address allocation → endpoint state → policy enforcement → inter-node forwarding → destination delivery.
Next section: Cilium Service Load Balancing and kube-proxy Replacement — ClusterIP, NodePort, eBPF socket load balancing, backend selection, Maglev, Direct Server Return and service troubleshooting.
Kubernetes Cilium Packet Path Lab
Investigate how Cilium connects Kubernetes workloads across nodes. Compare native routing with VXLAN and Geneve overlays, inspect endpoint and IPAM state, calculate encapsulation overhead, and troubleshoot inter-node connectivity failures.
Two-Node Cilium Networking Architecture
Follow traffic from Pod A on Node 1 across the underlay to Pod B on Node 2. Select any component for its engineering responsibilities.
Cilium Pod Networking Console
Network State Controls
Overlay overhead assumes IPv4 outer transport and a baseline 50-byte VXLAN or Geneve encapsulation. Geneve options, IPv6 outer headers, and additional encapsulation can increase overhead.
Pod IPAM and Routing Inspector
| Component | Network State | Status |
|---|
Inter-Node Packet Processing Pipeline
Cilium Packet Diagnostic Console
Engineering Interpretation
Production Engineering Notes
Cilium can operate using native routing or encapsulation-based routing. Native routing requires the surrounding network to provide appropriate reachability to pod prefixes. Overlay routing transports pod packets between nodes using tunnel encapsulation.
VXLAN and Geneve introduce additional headers. For a baseline IPv4 outer transport, the encapsulation overhead is commonly approximately 50 bytes. Real overhead depends on outer IP version, Geneve options, and other networking features.
IP address management is deployment-dependent. Cilium supports several IPAM approaches, including Kubernetes-based, cluster-pool, and cloud-specific modes. The simulator uses representative pod addresses rather than querying a real Kubernetes cluster.
Educational simulation. This lab models selected connectivity dependencies and assumes packets are not fragmented. A real deployment may use PMTU discovery, ICMP feedback, TCP MSS adjustment, or different encapsulation and forwarding mechanisms.
Cilium Service Load Balancing and kube-proxy Replacement
Cilium can implement Kubernetes Service connectivity using eBPF-based load balancing, replacing supported kube-proxy forwarding functions and enabling additional datapath optimisations.
Traditional Kubernetes Service implementations often rely on kube-proxy programming iptables or IPVS rules. With supported kube-proxy replacement configurations, Cilium maintains Service and backend state in eBPF maps and executes relevant load-balancing logic at kernel hooks.
The architecture supports Kubernetes Service types including ClusterIP, NodePort and LoadBalancer, subject to feature support and deployment configuration.
From Virtual Service Address to Application Backend
A Kubernetes Service provides a stable logical destination while its backend endpoints may change as pods scale, restart or move between nodes.
Cilium watches relevant Kubernetes Service and endpoint resources and maintains corresponding datapath state for supported service handling.
ClusterIP, NodePort and LoadBalancer
ClusterIP
Provides a stable virtual IP for in-cluster access. Supported Cilium eBPF load balancing selects a backend for the Service.
NodePort
Exposes a Service through a port on supported node addresses. External traffic can be handled through the configured Cilium NodePort datapath.
LoadBalancer
Integrates a Kubernetes Service with an external address provisioning mechanism. Cilium handles supported datapath forwarding; external IP allocation and advertisement depend on deployment integrations.
Architecture distinction: kube-proxy replacement does not automatically provision a public load balancer or advertise Service addresses to the external network. Address allocation, routing advertisement and external traffic delivery must be designed separately.
Compare Service Load-Balancing Mechanisms
Select a forwarding model to examine the different points at which Cilium can translate or direct Service traffic.
For supported local socket operations, Cilium can resolve a Service to a backend at an applicable socket hook. This can avoid some later packet-level translation work. Behaviour depends on the enabled socket load-balancing features.
Application → Socket Hook → Service Lookup → Backend Socket Destination
How Virtual Services Map to Real Endpoints
Cilium maintains datapath state for Service frontends and backend endpoints. A Service frontend identifies the logical destination; backend entries represent eligible application endpoints.
Connection-tracking and translation state help maintain consistent handling for supported flows. Backend changes must be reflected in the relevant datapath state as Kubernetes endpoint membership changes.
Random Selection and Maglev Consistent Hashing
Cilium supports configurable load-balancing algorithms in relevant datapath modes. Selection algorithms influence how new flows are distributed and how much backend reassignment occurs when membership changes.
| Algorithm | Principle | Engineering Consideration |
|---|---|---|
| Random | Chooses among eligible backends using the configured random selection mechanism. | Simple distribution; does not inherently provide consistent hashing across backend-set changes. |
| Maglev | Uses consistent hashing and lookup-table construction to select backends for supported flows. | Can reduce remapping when backends change; requires compatible mode and configuration. |
Operational principle: Maglev is not application-session replication. Consistent backend selection can reduce remapping, but a backend failure may still interrupt established application connections.
Designing the Forward and Return Traffic Paths
External Service traffic may use different translation and return-path strategies. These choices affect source-IP preservation, network routing, performance and troubleshooting.
| Mode | Forward Path | Return Path | Design Consideration |
|---|---|---|---|
| SNAT-based | Ingress node forwards to selected backend, applying relevant address translation. | Return traffic follows the translation-aware path required by the configured mode. | Source-address visibility and connection-tracking behaviour. |
| DSR | Ingress node directs traffic toward the selected backend while preserving the intended client-facing Service semantics. | Backend can return directly to the client in supported configurations. | Routing symmetry requirements differ; encapsulation or dispatch support and topology must be validated. |
DSR can reduce return-path forwarding through the ingress node, but it is not universally suitable. Network topology, client reachability, packet format and supported Cilium features determine whether it can be deployed successfully.
Moving Kubernetes Service Processing into eBPF
With supported kube-proxy replacement enabled, Cilium can implement Kubernetes Service forwarding functions that would otherwise be handled by kube-proxy.
Engineers must validate the required kernel capabilities, Service features, device selection, NodePort reachability and interactions with existing networking components before removing kube-proxy from an established cluster.
Diagnosing Cilium Service Failures
| Symptom | Investigation Area | Validation |
|---|---|---|
| ClusterIP unreachable | Service map state, backend eligibility, policy or routing. | Compare Kubernetes Service endpoints with Cilium service state. |
| NodePort unavailable externally | Device selection, firewall, routing, kube-proxy replacement configuration. | Test node reachability and external forwarding path. |
| Intermittent backend failures | Endpoint readiness, backend churn, connection state. | Correlate backend selection with pod readiness and events. |
| Incorrect source IP | SNAT, DSR or external traffic policy. | Inspect configured forwarding mode and observed source addresses. |
| Return traffic missing | Asymmetric routing, translation state or unsupported DSR topology. | Trace both forward and return paths. |
Service Connectivity Is a Datapath and Routing Problem
Cilium's eBPF service implementation connects Kubernetes Service definitions to kernel-level forwarding mechanisms. The Service frontend, backend membership, selection algorithm, translation state and return path must all operate correctly.
The central engineering model is Kubernetes Service → Cilium service state → eBPF lookup → backend selection → forwarding and return-path validation.
Next section: Cilium Network Policies and Zero Trust Security — endpoint identities, L3/L4/L7 enforcement, DNS-based policy, default-deny behaviour, policy troubleshooting and production security design.
eBPF Service Load Balancer Simulator
Explore how Cilium implements Kubernetes Service connectivity using eBPF maps and load-balancing programs. Investigate ClusterIP, NodePort, socket-level translation, backend selection, Maglev-style consistent hashing, and Direct Server Return.
Cilium eBPF Service Networking Architecture
Follow a client connection through the Cilium service translation layer to a selected backend. The return path changes according to the forwarding mode.
Service Load Balancing Console
Service Datapath State
Backend Availability
eBPF Service and Backend Map Inspector
| Backend | Endpoint | Availability |
|---|
Service Connection Processing Pipeline
eBPF Service Diagnostic Console
Engineering Interpretation
Production Engineering Notes
Cilium can replace kube-proxy for supported Kubernetes Service functionality using eBPF-based load balancing. The relevant programs and maps implement Service lookup, backend selection, and packet translation.
Socket-level load balancing may perform service translation at connection-related hooks. Packet-level load balancing operates at applicable datapath hooks. The exact behavior depends on Cilium configuration, workload type, and kernel support.
Maglev provides consistent hashing intended to reduce backend remapping during membership changes. The simulator uses a simplified deterministic hash for teaching rather than reproducing Cilium's complete Maglev lookup-table construction.
Direct Server Return requires a supported deployment and compatible return routing. SNAT and DSR should be evaluated against source-IP preservation, network topology, operational complexity, and service exposure requirements.
Educational simulation only. Backend health, hashing, packet translation, and return paths are simplified. The widget does not query Kubernetes or Cilium APIs and does not change any live cluster configuration.
Cilium Network Policies and Zero Trust Security
Cilium implements workload-aware security using endpoint identities, declarative network policies and supported eBPF enforcement mechanisms.
Rather than depending exclusively on fixed IP addresses, Cilium can derive security identities from Kubernetes labels and other supported endpoint metadata. Policies can then describe which workload groups may communicate, on which protocols and ports, and under what supported application-layer conditions.
This creates a foundation for Zero Trust network segmentation in Kubernetes, where connectivity is explicitly constrained and validated rather than assumed to be safe because workloads share a cluster.
Identity-Based Enforcement Between Kubernetes Workloads
Consider a frontend application that needs to reach an API service, while an unrelated workload should not have access to the same backend.
Cilium can evaluate the source and destination security context against applicable policy. Allowed traffic proceeds through the configured datapath; denied traffic is dropped or rejected according to the relevant enforcement mechanism.
Why Workload Identity Matters More Than a Fixed Pod IP
Kubernetes pods are dynamic. They are created, replaced, rescheduled and assigned new IP addresses. Security policies tied exclusively to individual addresses become difficult to maintain.
Cilium associates managed endpoints with security identities derived from relevant labels and metadata. Policies can reference those identities to express communication intent independently of an individual pod's current IP address.
Endpoint Labels
Relevant Kubernetes labels and metadata help determine the workload security identity.
Policy Selectors
Selectors describe source and destination workload groups that a policy applies to.
Kernel Enforcement
Supported datapath programs use identity and policy state to enforce network access decisions.
Engineering principle: Identity-based network policy does not replace application authentication, encryption or authorisation. Zero Trust requires controls at multiple layers.
Controlling Workload Reachability and Transport Ports
Layer 3 and Layer 4 policies constrain communication using workload identity, addresses, protocols and ports. For example, an API workload may accept TCP 8080 from selected frontend workloads while denying other ingress traffic once the relevant policy enforcement applies.
For selected endpoints and enforced directions, traffic that does not match an applicable allow rule is denied, subject to the complete policy configuration and Cilium's policy semantics.
Extending Security Beyond IP Addresses and Ports
Cilium supports application-layer policy for selected protocols and configurations, including HTTP-aware controls. L7 enforcement may involve Envoy proxy integration rather than being implemented entirely by packet-level eBPF instructions.
An application-layer policy can constrain requests using supported attributes such as HTTP methods and paths, helping reduce the scope of permitted API operations.
TLS consideration: Application-layer enforcement requires visibility into the relevant protocol content. Encryption and proxy termination design determine which HTTP attributes can be inspected.
Controlling External Destinations by Domain Name
Cilium supports DNS-aware egress policy in supported configurations. DNS responses can help maintain destination-IP information associated with permitted fully qualified domain names.
This can simplify policies for external services whose addresses change, but DNS-aware policy must be designed carefully because caching, shared IP addresses, DNS resolution paths and application behaviour influence the effective access boundary.
| Control | Purpose | Engineering Consideration |
|---|---|---|
| FQDN matching | Permit supported external destinations by DNS name. | DNS-derived IP state and cache lifecycle. |
| DNS proxy visibility | Observe and enforce supported DNS request behaviour. | DNS transport and resolver configuration. |
| Egress port restrictions | Limit permitted transport protocols and ports. | Required service connectivity and exceptions. |
| External identity handling | Apply supported policy to traffic outside the cluster. | Destination classification and routing context. |
Compare Common Zero Trust Traffic Scenarios
Select a scenario to explore how policy intent affects communication between Kubernetes workloads.
Selected frontend workloads are permitted to reach the backend API on TCP 8080 by an applicable ingress allow rule. Other policy constraints must also permit the communication.
Frontend Identity → API Policy → TCP 8080 Allowed
Building a Predictable Security Boundary
In Kubernetes, network-policy enforcement depends on which endpoints are selected and which traffic directions are subject to policy. Cilium supports multiple policy resources, including Kubernetes NetworkPolicy and Cilium-specific policy types.
Engineers should not assume that ordinary policy resources behave like a sequential firewall rule list. Applicable allow policies are generally composed, while explicit deny policies and supported policy tiers introduce additional semantics.
| Policy Concept | Meaning | Risk |
|---|---|---|
| Endpoint selection | Defines workloads affected by a policy. | Incorrect labels leave intended workloads unprotected. |
| Ingress isolation | Restricts incoming traffic for selected endpoints. | Missing required source rules break application connectivity. |
| Egress isolation | Restricts outgoing traffic for selected endpoints. | DNS, identity providers and external APIs may become unreachable. |
| Policy composition | Combines applicable policy resources according to supported semantics. | Assuming first-match firewall behaviour can lead to incorrect conclusions. |
| Explicit deny | Uses supported Cilium deny policy features. | Must account for documented precedence and feature behaviour. |
Deployment principle: Introduce restrictive policies with a validated application dependency map. Include DNS, monitoring, health checks and control-plane connectivity in the design.
Investigating Policy Denials and Unexpected Access
Policy troubleshooting begins by identifying the source endpoint, destination endpoint, security identities, selected policies and observed flow verdict.
Zero Trust Requires Identity, Enforcement and Verification
Cilium combines workload identities, declarative policy and supported eBPF enforcement mechanisms to implement Kubernetes network segmentation. Layer 7 and DNS-aware controls extend the policy model where supported, but require additional operational consideration.
The central engineering model is workload identity → applicable policy → kernel or proxy enforcement → flow verdict → operational verification.
Next section: Hubble Observability and eBPF Network Telemetry — flow events, service maps, policy verdicts, DNS visibility, distributed network troubleshooting and production monitoring architecture.
Cilium Zero Trust Policy Lab
Investigate how Cilium uses workload identities and eBPF policy enforcement to secure Kubernetes traffic. Test L3/L4 rules, L7 HTTP filtering, DNS-aware egress, and default-deny behavior across application workloads.
Kubernetes Identity-Based Security Architecture
Follow traffic from a frontend workload through Cilium policy enforcement to an internal API or external destination.
Cilium Security Policy Console
Security Enforcement Controls
Cilium Policy and Identity Inspector
| Policy Component | Configured Rule | Status |
|---|
Zero Trust Policy Evaluation Pipeline
Cilium Policy Diagnostic Console
Engineering Interpretation
Production Engineering Notes
Cilium associates workload endpoints with security identities derived from relevant labels. Identity-based policy is more resilient to changing pod IP addresses than policies relying exclusively on static addresses.
Cilium policies can express L3/L4 restrictions and supported L7 rules. L3/L4 enforcement is implemented through the eBPF datapath. L7 enforcement may require traffic redirection to a proxy, depending on the configured protocol and policy.
DNS-aware egress requires appropriate DNS rules and destination policy configuration. Allowing DNS resolution does not automatically authorize every subsequent connection to the resolved addresses.
Policy enforcement is directional. Ingress and egress rules apply to selected endpoints according to policy semantics. Default-deny behavior depends on endpoint policy selection and enforcement mode, not merely the presence of Cilium in a cluster.
Educational simulation only. The policy engine models a single simplified rule set. Real Cilium policy evaluation includes policy selection, directional enforcement, identity resolution, and potentially multiple interacting policy resources.
Hubble Observability and eBPF Network Telemetry
Hubble provides network-flow visibility, security-policy verdicts and service-level observability for supported Cilium-managed traffic.
Traditional Kubernetes troubleshooting often requires engineers to correlate pod logs, node packet captures, firewall counters and routing state. Hubble adds structured flow events derived from Cilium's datapath observability mechanisms, helping engineers identify which workloads communicated, what traffic was observed and whether policy enforcement permitted or denied it.
The Hubble architecture separates node-local event collection from optional cluster-wide aggregation and visualisation, allowing teams to investigate traffic without treating the observability system as a mandatory forwarding component.
From eBPF Datapath Events to Cluster-Wide Visibility
Hubble receives supported flow information from Cilium's datapath event infrastructure. Node-local Hubble capabilities expose flows for a managed node, while Hubble Relay can provide a cluster-wide view by connecting to node-local Hubble endpoints.
Hubble UI and CLI clients consume the available flow information to support investigation and service dependency analysis.
What a Hubble Flow Can Reveal
A Hubble flow record can include workload identities, source and destination information, transport protocol, observed traffic direction, verdict and relevant event metadata.
Depending on configuration and visibility, Hubble can also expose supported DNS, HTTP and other protocol-level observations.
Local Hubble, Relay and User Interfaces
Node-Local Hubble
Provides access to supported flow events associated with the Cilium-managed node, subject to buffer capacity and event availability.
Hubble Relay
Connects to node-local Hubble endpoints to expose cluster-wide flow information to compatible clients.
Hubble UI and CLI
Enable operators to inspect flow information, identify workload communication and explore service dependencies.
Engineering distinction: Hubble improves visibility into supported Cilium datapath events. It does not guarantee that every packet or application request is retained, nor does it replace packet captures, application tracing or comprehensive log collection.
Policy Verdicts, DNS Queries and Application Traffic
Hubble can help engineers correlate traffic observations with policy decisions and application connectivity symptoms. Select a flow category to examine a typical investigation.
A supported flow event indicates that traffic was forwarded at the observation point. Engineers should still validate end-to-end application delivery and return traffic.
Source Workload → Cilium Datapath → Forwarded Verdict → Destination
Understanding Which Applications Communicate
Service dependency visualisation helps reveal relationships between Kubernetes workloads and services. This is useful for application discovery, segmentation planning and identifying unexpected communication paths.
However, a dependency graph represents observed activity within the available visibility window. The absence of an observed flow does not prove that no dependency exists.
| Visibility Type | Engineering Value | Limitation |
|---|---|---|
| Source and destination workloads | Identify communicating namespaces and pods. | Depends on available identity and event metadata. |
| Policy verdicts | Investigate allowed and denied traffic. | Event context and enforcement semantics must be interpreted correctly. |
| DNS observations | Investigate name resolution and egress dependencies. | Visibility depends on DNS path and configuration. |
| HTTP observations | Inspect supported application-layer events. | Requires appropriate protocol visibility and proxy integration. |
| Service map | Visualise observed application relationships. | Not a complete inventory of all possible dependencies. |
Exporting Network Telemetry to Monitoring Platforms
Hubble can expose supported metrics for integration with Prometheus-based monitoring architectures. Available metrics and labels depend on enabled metric handlers and configuration.
Teams can use network-flow and policy-related metrics to detect changes in connectivity patterns, monitor drops and support operational dashboards.
Cardinality consideration: Highly granular labels can increase metric-series counts and monitoring cost. Production metric configuration should balance diagnostic value with storage, scrape and query overhead.
Interpreting Observability Data Correctly
Hubble's event stream is not a substitute for understanding the underlying datapath. Event buffering, filtering, aggregation, feature support and observation points affect which information is available.
| Limitation | Operational Impact | Mitigation |
|---|---|---|
| Finite event buffers | Events may be lost under sustained load. | Monitor event-loss indicators and size buffers appropriately. |
| Incomplete protocol visibility | Some application details are unavailable. | Validate enabled protocol and proxy features. |
| Aggregation gaps | Relay may not reach every node-local endpoint. | Check Relay connectivity and node availability. |
| Event interpretation | A forwarded event does not prove successful application processing. | Correlate with application logs, traces and response behaviour. |
| Retention limitations | Historical flows may not remain accessible. | Use supported external export or monitoring integrations where needed. |
Investigating Connectivity with Hubble
Start with the affected workload pair and time window. Filter by namespace, pod, verdict or protocol, then correlate observations with policy and routing state.
Observability Makes Kernel Networking Investigable
Hubble exposes supported Cilium datapath events through node-local and cluster-wide interfaces, enabling engineers to investigate workload communication, policy decisions and selected application protocols.
The core operational model is eBPF event generation → node-local Hubble → optional Relay aggregation → flow analysis → incident correlation.
Next section: Cilium BGP Control Plane and External Connectivity — BGP peerings, pod CIDR advertisements, Service VIP routing, external routers, route policy, multi-node availability and production troubleshooting.
Hubble Network Observability Explorer
Investigate Kubernetes networking using simulated Cilium Hubble flow telemetry. Inspect workload communications, policy verdicts, DNS activity, service dependencies, and datapath forwarding failures.
Kubernetes Service Dependency Map
Select workloads to inspect their dependencies, or select a flow record to animate its observed packet path.
Hubble Operations Console
Incident Analysis
Hubble observes Cilium datapath events and can expose flow verdicts, identities, protocols, and supported application-layer metadata. Available fields depend on configuration and visibility.
Live Hubble Flow Records
Select any flow to inspect its source, destination, protocol, verdict, and modeled drop reason.
| Flow | Source | Destination | Protocol | Verdict |
|---|
Selected Flow Inspector
Hubble Diagnostic Console
Production Engineering Notes
Hubble provides observability into network flows processed by Cilium. It can expose source and destination identities, forwarding verdicts, drop reasons, and supported protocol metadata.
Hubble Relay can aggregate flow information across nodes. Hubble UI provides graphical service dependency visualization. The exact scope of visibility depends on enabled components, datapath observation points, event availability, and configured flow filters.
A dropped flow is not always caused by NetworkPolicy. Routing failures, invalid destinations, unavailable services, and other datapath conditions may also produce observable failures.
Engineers should correlate Hubble observations with Cilium endpoint state, Kubernetes Service resources, policy definitions, routing tables, and application logs.
Educational simulator. All flow records are generated locally in the browser. No live Kubernetes cluster, Cilium agent, or Hubble Relay is contacted.
Cilium BGP Control Plane and External Connectivity
Cilium's BGP Control Plane allows supported Kubernetes nodes to establish BGP sessions with external routers and advertise selected network prefixes, connecting cluster networking to enterprise and data centre routing infrastructure.
While Cilium's eBPF datapath forwards workload traffic, the BGP Control Plane provides a mechanism for distributing supported routing information beyond the Kubernetes cluster.
This separation is fundamental: BGP distributes reachability information; the Linux and Cilium datapaths forward the actual packets.
Connecting Kubernetes Nodes to the External Routing Fabric
In a typical design, Cilium-managed Kubernetes nodes establish BGP sessions with upstream routers. The nodes advertise selected pod CIDRs or supported Service address prefixes, subject to configuration and advertisement policies.
External routers then use BGP routes to determine where traffic should be forwarded toward the Kubernetes cluster.
Peers, Advertisements and Route Selection
BGP Peer Configuration
Defines supported peer relationships between Cilium-managed nodes and external routers, including peer addresses and AS numbers.
Route Advertisements
Controls which supported pod CIDRs and Service address prefixes are advertised to configured peers.
External Routing Fabric
Receives advertisements and selects forwarding paths according to its own BGP policy and routing configuration.
Critical distinction: The Cilium BGP Control Plane advertises reachability. It is not a general-purpose replacement for the Linux routing stack or the Cilium datapath, and BGP advertisements alone do not guarantee working pod-to-pod forwarding.
Making Kubernetes Pod Networks Reachable
In supported native-routing architectures, pod CIDR advertisements can inform external routers about the network prefixes associated with Kubernetes nodes.
For example, Node A may advertise 10.0.1.0/24 while Node B advertises 10.0.2.0/24. The upstream fabric can then forward traffic toward the node responsible for the relevant destination prefix.
Advertising Kubernetes Services to External Routers
Cilium can advertise supported Service addresses through its BGP Control Plane. This enables external routers to learn routes toward Service VIPs without relying solely on static route configuration.
Service address advertisement and Service load balancing are separate functions: BGP delivers traffic toward a node, while the configured service datapath selects and forwards to an eligible backend.
| Address Type | Purpose | Engineering Consideration |
|---|---|---|
| Pod CIDR | Advertise supported pod network reachability. | Native-routing design and prefix ownership. |
| LoadBalancer VIP | Advertise supported external Service address reachability. | Service allocation, backend placement and external traffic policy. |
| ExternalIP | Advertise supported Service ExternalIP addresses. | Address ownership and external routing design. |
| ClusterIP | Advertise supported ClusterIP addresses where configured and supported. | Service CIDR reachability and intended exposure boundaries. |
Security consideration: Advertising a Service prefix does not automatically make that Service safe for external consumption. Routing reachability must be combined with firewall policy, workload security and appropriate Service exposure controls.
Compare BGP Routing Scenarios
Select a scenario to examine the control-plane behaviour and forwarding implications.
A Kubernetes node advertises its supported pod prefix to an upstream BGP peer. The external router installs a route according to its policy and forwards matching traffic toward the node.
Node Pod CIDR → BGP Advertisement → External RIB/FIB → Node Forwarding
Multiple Next Hops for Service Reachability
When supported Service prefixes are advertised from multiple nodes, an upstream routing fabric may install multiple equal-cost next hops, subject to BGP policy and router capabilities.
ECMP can distribute traffic across eligible next hops, but the resulting traffic behaviour depends on hashing, backend availability, Service external traffic policy and the configured Cilium load-balancing mode.
What Happens When a Node or BGP Session Fails?
BGP failure recovery depends on how the failure is detected, when routes are withdrawn or invalidated, and how quickly external routers update their forwarding tables.
A failed node, lost BGP session or changed Service eligibility may affect the available routes. However, BGP session state and application health are not the same thing, and not every application failure automatically causes a route withdrawal.
| Failure | Potential Routing Impact | Investigation |
|---|---|---|
| BGP peer failure | Routes learned from the affected session may be withdrawn after failure detection. | Inspect peer state, timers and route tables. |
| Kubernetes node failure | Node-originated advertisements may become unavailable. | Check node health and upstream route convergence. |
| Service backend failure | Service forwarding eligibility changes; route behaviour depends on advertisement policy. | Compare Service endpoints with advertised routes. |
| Underlay failure | BGP may remain established or fail depending on topology. | Validate next-hop reachability and forwarding path. |
Investigating Missing BGP Routes and Broken External Connectivity
Begin with the BGP session state and configured advertisements. Then validate whether the external router received, accepted and installed the expected route.
BGP Connects Kubernetes Reachability to the Network Fabric
Cilium's BGP Control Plane provides a mechanism for advertising supported pod and Service prefixes to external routing infrastructure. External routers determine how those advertisements influence forwarding, while the Cilium datapath handles relevant workload and Service traffic.
The essential engineering model is prefix ownership → BGP advertisement → external route selection → next-hop forwarding → Cilium datapath → destination workload.
Next section: Cluster Mesh, Encryption and High Availability — multi-cluster connectivity, shared identities, service discovery, WireGuard/IPsec encryption, failure domains and production resilience engineering.
Cilium BGP Routing Simulator
Explore how Cilium's BGP control plane exchanges routing information with external network devices. Investigate peer establishment, prefix advertisement, service reachability, and route withdrawal.
Kubernetes BGP Control Plane Architecture
Two Cilium nodes establish eBGP sessions with an external leaf router. Each node advertises its configured prefixes. The router installs accepted routes into its routing table.
BGP Routing Engineering Console
BGP Control Plane State
External Router BGP Routing Table
| Prefix | Next Hop | Origin | Status |
|---|
Routes appear after successful BGP advertisement and acceptance.
BGP Session and Route Processing
BGP Diagnostic Console
Engineering Interpretation
Production Engineering Notes
Cilium's BGP control plane enables supported Kubernetes networking prefixes to be advertised to external BGP routers. Configuration determines which peers, address families, pod CIDRs, and Service addresses are advertised.
BGP peering does not automatically guarantee end-to-end workload connectivity. External route acceptance, next-hop reachability, Cilium datapath state, Service forwarding, and network policy must also be considered.
A BGP session may be Established while a required prefix is absent because advertisement configuration, address eligibility, or external routing policy prevents the route from being installed.
In production, engineers should inspect Cilium BGP peer status, advertised routes, external router RIB/FIB state, Kubernetes Service configuration, and datapath reachability before concluding that a BGP control-plane problem is the sole cause of an outage.
Educational simulator only. The lab models simplified eBGP peering and route advertisement. It does not implement the complete BGP finite state machine, route selection algorithm, or live Cilium configuration.
Cluster Mesh, Encryption and High Availability
Cilium Cluster Mesh extends supported workload connectivity and security identity capabilities across multiple Kubernetes clusters, enabling multi-cluster networking, global service discovery and policy-aware communication.
In production, multi-cluster architecture introduces additional failure domains, network paths and security boundaries. Engineers must design cluster connectivity, address allocation, service availability, encryption and operational recovery as a coordinated system.
Cilium also supports transparent node-to-node traffic encryption using supported WireGuard or IPsec configurations, providing protection for relevant workload traffic in transit.
Connecting Kubernetes Clusters Across Regions and Networks
Cluster Mesh allows supported Cilium-managed clusters to exchange the metadata and connectivity information needed for cross-cluster workload communication.
Each cluster retains its own Kubernetes control plane and local networking infrastructure. Cluster Mesh adds mechanisms for exchanging relevant identities, endpoint information and global service state.
Identity, Endpoint Discovery and Global Service State
Cluster Identity
Participating clusters require appropriate cluster identifiers and compatible identity allocation arrangements to support cross-cluster security semantics.
Shared State Exchange
Cluster Mesh exchanges relevant endpoint, identity and service information through its supported control-plane mechanisms.
Global Services
Supported global service configuration can make eligible backends from multiple clusters available through a coordinated service abstraction.
Design requirement: Pod address ranges must be planned to avoid unsupported overlap. Cross-cluster datapath reachability is a separate requirement from successful Cluster Mesh control-plane connectivity.
Distributing Application Connectivity Across Clusters
Cilium global services can support multi-cluster service discovery and backend selection in compatible configurations. This allows selected services to expose eligible backends across participating clusters.
Engineers must distinguish between a global service definition, the current set of healthy or eligible backends, and the routing path used to reach those backends.
| Capability | Purpose | Engineering Consideration |
|---|---|---|
| Global service discovery | Expose supported service backends across clusters. | Matching service configuration and control-plane health. |
| Backend selection | Choose among eligible local and remote endpoints. | Traffic distribution, latency and failure behaviour. |
| Service affinity | Prefer local or remote backends where supported and configured. | Capacity and resilience tradeoffs. |
| Cross-cluster reachability | Deliver traffic to selected remote endpoints. | Routing, firewall policy, MTU and encryption. |
WireGuard and IPsec for Workload Traffic Protection
Cilium supports transparent encryption using supported WireGuard and IPsec datapath configurations. Encryption protects relevant traffic in transit between participating nodes, subject to feature and deployment limitations.
Encryption design must account for key distribution, kernel support, packet overhead, routing compatibility and operational recovery.
| Technology | Architecture | Operational Consideration |
|---|---|---|
| WireGuard | Uses WireGuard-based encrypted tunnels in supported Cilium configurations. | Kernel compatibility, peer state, keys and MTU. |
| IPsec | Uses supported IPsec-based encryption for relevant node-to-node traffic. | Key management, encryption overhead and compatibility. |
| Application TLS | Encrypts application communication at a higher layer. | Certificates, endpoint identity and application termination. |
Security principle: Network-layer encryption does not automatically provide application-level authentication or authorisation. Use workload identity, network policy and application security controls together.
Compare Connectivity and Failure Conditions
Select a scenario to examine the expected architectural behaviour.
Participating clusters exchange supported Cluster Mesh state and have working network reachability. Eligible remote service backends can receive traffic according to the configured global service behaviour.
Cluster Mesh State → Backend Discovery → Cross-Cluster Forwarding
Designing for Independent Cluster and Network Failures
Multi-cluster deployment can improve resilience, but it also introduces additional dependencies. Cluster Mesh control-plane availability, inter-cluster routing, encryption state and application backend health must be evaluated separately.
Understanding Control-Plane and Data-Plane Separation
A loss of Cluster Mesh state exchange does not necessarily mean that every established cross-cluster packet flow stops immediately. Conversely, a healthy Cluster Mesh control plane does not prove that remote workload traffic is deliverable.
| Failure | Potential Impact | Investigation |
|---|---|---|
| Cluster Mesh state exchange failure | Remote identity, endpoint or service state may become stale or unavailable. | Check Cluster Mesh status and control-plane connectivity. |
| Inter-cluster network failure | Remote workloads become unreachable despite valid service metadata. | Trace underlay routes and firewall policy. |
| Encryption peer failure | Protected traffic may fail between affected nodes. | Inspect encryption state, keys and peer reachability. |
| Remote backend failure | Eligible backend set changes; service continuity depends on remaining capacity and configuration. | Compare service endpoints and observed traffic. |
| Address overlap | Ambiguous routing or unsupported cross-cluster reachability. | Audit pod and service address plans. |
Validating Cluster Mesh and Encrypted Connectivity
Start with cluster identities and Cluster Mesh health. Then validate remote endpoint discovery, inter-cluster routes, policy and encryption state.
Multi-Cluster Resilience Depends on More Than Connectivity
Cilium Cluster Mesh can extend workload identity, service discovery and supported networking capabilities across Kubernetes clusters. WireGuard and IPsec provide additional transport security options, while high availability requires deliberate handling of control-plane, datapath and application failures.
The essential engineering model is cluster identity → shared state → remote endpoint discovery → routed and secured forwarding → failure recovery.
Next section: Production Troubleshooting and Performance Engineering — packet drops, MTU failures, route selection, eBPF verifier issues, map pressure, connection tracking, service failures and systematic incident investigation.
Cilium Multi-Cluster Failure Investigation Lab
Investigate cross-cluster Kubernetes networking using Cilium Cluster Mesh. Examine global service connectivity, remote identity synchronization, encryption transport, cluster availability, and failover behavior.
Multi-Cluster Cilium Architecture
Two Kubernetes clusters share selected service and identity information through Cluster Mesh. Application traffic traverses a separate inter-cluster network.
Cluster Mesh Engineering Console
Cluster Health and Control Plane
Cluster Mesh State Inspector
| Component | State | Status |
|---|
Multi-Cluster Request Processing
Cluster Mesh Diagnostic Console
Engineering Interpretation
Production Engineering Notes
Cilium Cluster Mesh enables multi-cluster connectivity and supported cross-cluster service discovery. Cluster identities, endpoint information, and global service configuration influence how workloads communicate.
The Cluster Mesh control plane must be distinguished from the data plane. Loss of synchronization can prevent new remote endpoint or identity updates, while existing forwarding state may continue temporarily depending on the failure and deployed configuration.
Cross-cluster encryption requires compatible node connectivity, key material, and Cilium configuration. WireGuard and IPsec are alternative encryption mechanisms, not simultaneous requirements.
Global service failover depends on endpoint health visibility, service annotations or other configuration, cluster reachability, and the selected traffic distribution behavior.
Educational simulation only. The model simplifies identity synchronization, service failover, and encryption dependencies. It does not represent a complete Cluster Mesh implementation or modify any live Kubernetes cluster.
Cilium Production Troubleshooting and Performance Engineering
Reliable Cilium operations require engineers to investigate Kubernetes state, eBPF datapath behaviour, network routing, policy enforcement, service forwarding and Linux kernel resources as one connected system.
Production incidents rarely present themselves as a single obvious eBPF failure. A broken application request may originate from an IPAM problem, an incorrect security identity, a missing Service backend, an unreachable pod CIDR, an MTU mismatch or exhausted connection-tracking resources.
The objective is to develop a repeatable investigation process that moves from the observed application symptom to the exact failing component.
Following a Failed Packet Through the Cilium Stack
When a workload cannot communicate, the most effective approach is to reconstruct the intended packet path and validate each forwarding decision.
The architecture below shows the major components involved in a typical Cilium incident investigation.
Classifying the Problem Before Changing Configuration
Control Plane
Cilium agent health, Kubernetes resource synchronisation, IPAM allocation, endpoint identities and BGP state.
Data Plane
eBPF program attachment, map lookups, packet forwarding, connection tracking, policy decisions and routing.
Infrastructure
Underlay reachability, MTU, cloud firewall rules, kernel support, network interfaces and resource pressure.
Operational rule: Do not restart Cilium agents or modify cluster-wide network configuration before collecting the evidence needed to distinguish a local endpoint issue from a systemic datapath failure.
Verifier Rejections, Attachment Errors and Kernel Compatibility
eBPF programs must pass kernel verification before they can be loaded. The verifier checks properties such as memory safety and valid program behaviour. A rejected program may prevent the intended datapath feature from becoming active.
Other failures may arise from missing kernel capabilities, unsupported helpers, incompatible program types or errors attaching programs to the required hooks.
| Failure Type | Potential Symptom | Investigation |
|---|---|---|
| Verifier rejection | eBPF program cannot load. | Inspect verifier output and agent logs. |
| Kernel capability mismatch | Required datapath feature unavailable. | Validate kernel and Cilium compatibility. |
| Program attachment failure | Traffic not processed at the intended hook. | Inspect loaded programs and attachment points. |
| Map creation failure | Datapath state unavailable or incomplete. | Inspect resource limits and agent logs. |
When Kernel State Becomes a Performance Bottleneck
Cilium relies on supported eBPF maps for connection tracking, service state, endpoint information and other datapath functions. High connection churn or insufficient resource sizing can create pressure on relevant state tables.
Symptoms may include failed new connections, increased drop events or inconsistent behaviour under load.
Increasing map sizes without understanding workload behaviour can consume additional kernel memory and may not resolve the underlying problem. Capacity planning should use observed connection rates, concurrent flow counts and resource utilisation.
Diagnose Common Production Failures
Select a failure scenario to review the likely investigation path.
A workload cannot reach its destination because applicable Cilium policy does not permit the flow. Inspect endpoint identities, policy selectors, ingress and egress enforcement, and Hubble drop verdicts.
Endpoint Identity → Policy Evaluation → Drop Event → Rule Correction
Why Small Packets Work but Large Packets Fail
MTU problems are common in overlay and encrypted networking because encapsulation adds packet overhead. A workload may successfully exchange small packets while larger packets fail or experience fragmentation-related issues.
Routing failures can produce similar symptoms but require a different investigation. Engineers must determine whether packets are being dropped because of size, missing routes, unreachable next hops or policy.
| Symptom | Possible Cause | Validation |
|---|---|---|
| Small requests work, large requests fail | MTU or path-MTU discovery issue. | Test packet sizes and inspect tunnel overhead. |
| Same-node works, cross-node fails | Underlay route, tunnel or firewall problem. | Compare node and pod connectivity. |
| Only external traffic fails | BGP, NAT, Service or upstream routing issue. | Trace forward and return paths. |
| Intermittent cross-cluster failure | Remote route, encryption or Cluster Mesh dependency. | Validate each cluster and inter-cluster path. |
Optimising Throughput, Latency and Datapath Efficiency
eBPF can reduce some networking overhead by processing traffic at supported kernel hooks, but performance depends on workload patterns, enabled features, kernel implementation and hardware capabilities.
Optimisation requires measurement rather than assuming that every eBPF feature improves every workload.
| Area | Potential Optimisation | Tradeoff |
|---|---|---|
| Service load balancing | Use supported efficient backend selection and socket handling. | Feature compatibility and workload behaviour. |
| Routing mode | Evaluate native routing versus encapsulation. | Underlay complexity and MTU overhead. |
| Connection tracking | Size maps using measured flow demand. | Kernel memory consumption. |
| Policy enforcement | Minimise unnecessary complexity and validate proxy requirements. | Security objectives must remain intact. |
| Observability | Choose appropriate event and metric collection levels. | Diagnostic coverage versus resource overhead. |
Performance principle: Benchmark representative traffic patterns before and after a configuration change. Measure latency, throughput, packet loss, CPU utilisation and application-level behaviour rather than relying on synthetic throughput alone.
A Repeatable Cilium Incident Investigation Process
Use a layered investigation sequence to identify the failing component and minimise unnecessary changes.
Understanding Cilium as a Complete Networking System
Cilium is more than a Kubernetes CNI plugin. It combines programmable eBPF datapath processing, workload identity, network policy, Service load balancing, routing integrations, observability and supported multi-cluster capabilities.
Reliable production operations require engineers to understand how these components interact and how failures propagate between them.
The complete engineering model is Kubernetes intent → Cilium control plane → endpoint and eBPF state → packet forwarding → policy enforcement → telemetry → operational verification.
Guide 21 complete: You have covered Cilium architecture fundamentals, eBPF datapath processing, Kubernetes pod networking and IPAM, Service load balancing, Zero Trust policies, Hubble observability, BGP connectivity, Cluster Mesh, encryption and production troubleshooting.
Cilium Network Incident Investigation Engine
Investigate production-style Kubernetes network failures using Cilium datapath state, eBPF maps, policy verdicts, routing information, DNS diagnostics, MTU analysis, and simulated Hubble observability. Identify the fault, apply a targeted repair, and verify recovery.
Cilium Production Datapath Architecture
Follow traffic through a Kubernetes application path. Each diagnostic layer can independently affect end-to-end connectivity.
Incident Response Console
Diagnostic Commands
Network State Comparison Engine
| Subsystem | Baseline | Current State |
|---|
Incident Processing Pipeline
Production Diagnostic Terminal
Root Cause and Recovery Analysis
Production Engineering Notes
Cilium production troubleshooting requires separating endpoint health, eBPF service translation, network policy, routing, DNS resolution, and MTU behavior. A successful control-plane operation does not guarantee that packets traverse the datapath successfully.
Hubble can provide useful flow-level evidence, including policy verdicts and supported drop reasons. However, engineers should correlate telemetry with Cilium endpoint state, Service maps, Kubernetes resources, kernel routes, and underlying network connectivity.
MTU failures are especially important in overlay environments. Encapsulation consumes additional packet overhead, and oversized packets may fail if fragmentation or Path MTU Discovery cannot resolve the mismatch.
The correct operational response is to identify the failing subsystem before applying a targeted change. Restarting components or broadly disabling security controls can conceal the underlying problem.
Educational incident simulator. Diagnostic outputs are illustrative, not live command results. The network state, repairs, and packet tests operate entirely inside this browser widget.
- AI Model Serving Performance Engineering Explained - November 20, 2026
- NVIDIA InfiniBand Networking Architecture Explained - November 20, 2026
- Azure Kubernetes Service Networking Architecture Explained - November 20, 2026
