Cilium eBPF Networking Architecture Guide

Cilium and eBPF Networking Architecture Explained

NETWORK INSIGHT · THE VISUAL AGE ENGINEERING GUIDE 21 / ARCHITECTURE INTRODUCTION
Kubernetes Networking · Linux Kernel · eBPF · Cloud Native Security

Cilium and eBPF Networking Architecture Explained

Modern Kubernetes networking is increasingly implemented through programmable Linux kernel datapaths. Cilium uses extended Berkeley Packet Filter (eBPF) to deliver container networking, service load balancing, identity-aware security and deep network observability.

Instead of relying exclusively on traditional Linux networking mechanisms such as iptables, Cilium can attach verified eBPF programs to supported kernel hooks, allowing network decisions to occur close to where packets and socket operations are processed.

This guide explores the relationship between Kubernetes workloads, the Cilium control plane, eBPF programs and maps, the Linux kernel networking stack, Hubble observability and external network connectivity.

Kubernetes CNI Linux eBPF TC / XDP / Socket Hooks eBPF Maps Identity-Based Security kube-proxy Replacement Hubble BGP Control Plane Cluster Mesh
CILIUM / eBPF — LAYERED NETWORK ARCHITECTURE ANIMATED PACKET PATH
KUBERNETES WORKLOAD LAYER Application Pod A 10.0.1.10 / Endpoint Identity Application Pod B 10.0.1.20 / Endpoint Identity Kubernetes Service ClusterIP / Backend Selection External Network Node / BGP / Gateway CILIUM USER-SPACE CONTROL PLANE Cilium Agent Endpoint / Policy / Datapath Cilium Operator Cluster-Level Management Hubble Flow Events / Visibility eBPF Map State Identity / Service / CT LINUX KERNEL / CILIUM eBPF DATAPATH TC / Socket Hooks eBPF Program Execution Policy Enforcement Identity / L3-L4 / L7 Integration Service Load Balancing Backend / CT / NAT State Forwarding Native / Overlay / NIC KERNEL EXECUTION OBSERVABILITY EVENTS
Figure 1 — Conceptual Cilium architecture. Control-plane components configure and manage networking state; supported eBPF programs execute in kernel contexts. Actual packet traversal depends on datapath mode, hook selection, kernel capabilities and workload location. Hubble consumes flow observability events rather than forwarding packets.
Interactive Architecture Perspectives

Explore Four Cilium Engineering Domains

Select an engineering domain to examine how Cilium connects Kubernetes networking, Linux kernel execution and operational visibility.

eBPF Packet Forwarding

Cilium attaches supported eBPF programs to Linux kernel hooks to implement packet processing, policy checks, forwarding and service connectivity. The precise path depends on the configured Cilium datapath.

Pod → Kernel Hook → eBPF Program → Policy / Routing → Destination
Architecture Fundamentals

The Three Layers of Cilium Networking

LAYER 01
Kubernetes Workloads

Pods, services and namespaces define application endpoints and desired connectivity. Kubernetes resources provide the inputs used by Cilium to configure networking behaviour.

LAYER 02
Cilium Control Plane

The Cilium agent manages endpoint state, security identities, service information and datapath configuration. The operator performs supported cluster-level management tasks.

LAYER 03
Linux eBPF Datapath

Verified eBPF programs execute at supported kernel hooks, using maps and kernel helpers to make forwarding, service and security decisions.

Packet Processing Model

From Kubernetes Pod to Network Destination

The exact processing sequence differs between same-node traffic, cross-node traffic, service traffic and external network flows. The following sequence is a simplified conceptual model.

01 Pod Generates Traffic Application initiates communication
02 Kernel Hook Supported eBPF attachment point
03 Policy Decision Identity and security evaluation
04 Service / Route Backend selection or forwarding
05 Destination Pod, node or external network
Production Engineering Perspective

Why Cilium Changes Kubernetes Networking

Cilium introduces a programmable networking model where supported forwarding and security functions can be implemented through eBPF rather than relying exclusively on conventional packet-processing mechanisms.

This provides opportunities for efficient service handling, workload identity-based security, detailed flow visibility and integration with external routing systems.

However, eBPF does not remove the need to understand the underlying Linux network stack. Engineers still need to investigate routing, MTU, conntrack state, kernel compatibility, endpoint identities, policy behaviour and packet drops.

Core engineering principle: Cilium's user-space components configure the desired networking behaviour, while supported eBPF programs enforce and implement datapath decisions inside the Linux kernel. Successful troubleshooting requires visibility into both layers.

Continue to Section 1 — Cilium and eBPF Architecture Fundamentals

Linux kernel hooks, eBPF programs and maps, Cilium agent, operator, endpoint identities and the Kubernetes CNI lifecycle.

EXPLORE SECTION 01 →
Network Insight · Engineering Knowledge Hub
GUIDE 21 / ANIMATED TOC V2
Explore the Architecture · Understand the Datapath

Cilium and eBPF Networking Architecture

Explore the Linux kernel datapath, Kubernetes networking, eBPF service load balancing, Zero Trust policies, Hubble observability, BGP routing, Cluster Mesh and production troubleshooting through nine interconnected engineering sections.

09Engineering Sections
eBPFProgrammable Datapath
L3–L7Security & Visibility
PROGRAMMABLE NETWORKING · IDENTITY · SECURITY · OBSERVABILITY
Interactive Guide Navigation

Explore All Nine Engineering Sections

Select any architecture domain to jump directly to its editorial section.

Sections reached 0 / 9
NETWORK INSIGHT · THE VISUAL AGE ENGINEERING GUIDE 21 · CILIUM & eBPF · ANIMATED TOC V2
Engineering Guide 21 · Editorial Section 01

Cilium and eBPF Architecture Fundamentals

Cilium is a cloud-native networking, security and observability platform that uses extended Berkeley Packet Filter (eBPF) to execute programmable networking functions inside the Linux kernel.

Its architecture connects Kubernetes configuration and workload identity to kernel-level packet processing. The Cilium agent interprets cluster state, manages endpoints and policy, and configures the eBPF datapath. The Linux kernel then executes verified programs at supported attachment points.

Understanding this separation between control-plane orchestration and kernel datapath execution is essential for engineering, securing and troubleshooting Cilium deployments.

01.1 / Layered Architecture

How Kubernetes, Cilium and the Linux Kernel Work Together

A Cilium-managed Kubernetes node contains workload endpoints, user-space management components and kernel-resident eBPF programs and maps. The components collaborate, but they do not all execute in the packet path.

CILIUM ARCHITECTURE — CONTROL PLANE AND KERNEL DATAPATH KUBERNETES WORKLOAD AND API LAYER Application Pods Namespaces / Labels / Endpoints Kubernetes API Services / Policies / Node State CNI Lifecycle Endpoint Provisioning USER-SPACE CONTROL PLANE Cilium Agent Endpoint / Policy / Datapath State Cilium Operator Cluster-Level Coordination Program Loader Load / Attach / Manage eBPF LINUX KERNEL EXECUTION LAYER Kernel Hooks TC / XDP / Socket / Cgroup Verified eBPF Programs Forwarding / Policy / Services eBPF Maps Identity / Service / CT / Policy CONTROL PLANE CONFIGURES STATE · VERIFIED eBPF PROGRAMS EXECUTE IN THE KERNEL
Figure 1 — Conceptual Cilium architecture. Component interactions are simplified; actual program loading, attachment and endpoint provisioning vary by deployment.
01.2 / eBPF Execution Model

From eBPF Source Code to Kernel Execution

eBPF programs are commonly compiled from C or other supported languages into eBPF bytecode. A privileged loader submits the program to the kernel, where the verifier checks program safety and permitted execution behaviour before it can be loaded.

On supported platforms, just-in-time compilation can translate verified eBPF bytecode into native machine instructions. The resulting program executes when its configured kernel hook is triggered.

01SourceProgram implementation
02CompileeBPF bytecode
03VerifyKernel safety checks
04Load / AttachProgram at supported hook
05ExecuteKernel event or packet

Important distinction: The verifier does not establish that a networking policy is operationally correct. It checks whether a program satisfies kernel verification constraints. Network behaviour must still be validated through policy testing, packet-path inspection and observability.

01.3 / Linux Attachment Points

Where eBPF Programs Interact with Networking

Attachment PointTypical UseEngineering Consideration
XDPEarly packet processing on supported network devices.Runs before much of the conventional networking stack; feature and driver support matter.
TCIngress and egress packet processing.Commonly used for endpoint forwarding, policy and service operations.
Socket / cgroup hooksSocket-level service and connectivity decisions.Can implement selected service translation and policy functions before packet forwarding.
Tracing hooksKernel event and diagnostic instrumentation.Useful for observability; not every tracing hook is a forwarding hook.

The exact attachment points used by Cilium depend on the configured datapath, enabled features, Linux kernel version and network-device capabilities. XDP, TC and socket hooks should not be treated as mandatory sequential processing stages for every packet.

01.4 / eBPF Maps

How Programs Share Networking State

eBPF maps are kernel-managed data structures that allow supported programs and user-space components to exchange and maintain state. Cilium uses maps for functions such as endpoint identity lookup, service backend selection, connection tracking and policy-related decisions.

CONCEPTUAL eBPF MAP LOOKUPS ENDPOINT IDENTITY: Endpoint IP / metadata → Security identity SERVICE LOOKUP: Service IP + Port → Backend selection data CONNECTION TRACKING: Flow tuple → Connection state POLICY: Identity / protocol / port context → Applicable enforcement decision NOTE: Actual Cilium map schemas, keys and values depend on version and feature.

Map capacity, lookup performance, lifecycle management and state consistency are important operational considerations. Stale or exhausted map state can contribute to connectivity failures and should be investigated alongside routes and security policy.

01.5 / Cilium Agent and Operator

Separating Node-Level and Cluster-Level Responsibilities

COMPONENT 01
Cilium Agent

Typically deployed on each managed node. Coordinates endpoint state, identities, policy, service information and datapath configuration.

COMPONENT 02
Cilium Operator

Performs supported cluster-level functions such as coordinating certain IPAM, identity and resource-management operations, depending on configuration.

COMPONENT 03
Linux Kernel Datapath

Executes loaded eBPF programs and maintains relevant map state, allowing supported forwarding decisions to occur without a user-space round trip for every packet.

Control-plane versus data-plane principle: An existing kernel datapath may continue processing supported traffic during some user-space control-plane interruptions, but new endpoint provisioning, policy updates and other operations can be affected. Behaviour depends on the failure and configuration.

01.6 / Kubernetes CNI Lifecycle

How a Pod Becomes a Cilium-Managed Endpoint

When Kubernetes schedules a pod, the container runtime invokes the configured CNI integration to establish networking. Cilium participates in provisioning the endpoint, configuring the relevant networking interfaces and preparing the datapath for the workload.

CONCEPTUAL POD NETWORK PROVISIONING 1. Kubernetes schedules a Pod. 2. Container runtime creates the required network namespace. 3. Configured CNI integration runs. 4. Endpoint networking is prepared: - Interface configuration - IP addressing - Routes / datapath integration 5. Cilium manages endpoint state and resolves security identity. 6. Applicable policy and service state are configured. 7. eBPF programs and maps support packet processing. 8. Endpoint becomes ready according to Kubernetes and Cilium state. Actual ordering and implementation vary by Cilium version and mode.

Troubleshooting pod connectivity therefore requires examining both Kubernetes lifecycle state and Cilium endpoint state. A running container is not sufficient evidence that all networking configuration and policy has converged.

01.7 / Security Identities

Moving Beyond IP-Based Network Security

Cilium uses security identities derived from relevant workload labels and metadata. These identities allow policy to describe relationships between groups of workloads rather than relying exclusively on fixed IP addresses.

This is especially useful in Kubernetes environments where pods are frequently recreated and IP addresses change. Identity-based policy can remain aligned with workload intent as endpoint addresses evolve.

CONCEPTUAL IDENTITY-BASED POLICY WORKLOAD A: namespace: frontend app: web WORKLOAD B: namespace: backend app: api INTENDED POLICY: Allow frontend → backend TCP destination port 8080 DENY: Other ingress traffic to backend when applicable policy selects the destination and no allow rule permits that traffic. Cilium evaluates supported policy using endpoint identity and relevant traffic context.

Policy semantics depend on the selected endpoints, direction, protocol, policy resources and configured enforcement mode. Identity-aware security complements, rather than automatically replaces, application authentication and authorisation.

01.8 / Production Validation

What Engineers Should Verify Before Deployment

✓Confirm supported kernel version and required eBPF capabilities.
✓Validate Cilium version and Kubernetes compatibility.
✓Confirm CNI installation and endpoint readiness.
✓Inspect effective service and policy configuration.
✓Validate pod-to-pod, pod-to-service and external connectivity.
✓Review eBPF map capacity and connection-tracking behaviour.
✓Confirm routing, MTU and overlay or native datapath settings.
✓Test failure recovery, upgrades and observability coverage.
ILLUSTRATIVE CILIUM DIAGNOSTICS # Cilium status (via Cilium CLI) cilium status # Cilium-managed endpoints cilium-dbg endpoint list # Service state cilium-dbg service list # eBPF map state cilium-dbg bpf map list # Recent datapath events cilium-dbg monitor # Policy state cilium-dbg policy get NOTE: cilium-dbg commands are typically run inside a Cilium agent container or equivalent diagnostic environment. CLI availability and syntax can vary by release and installation.
01.9 / Engineering Summary

The Foundation of Cilium Networking

Cilium combines Kubernetes-aware control-plane management with programmable Linux kernel execution. The agent and operator coordinate supported networking state, while verified eBPF programs and maps implement selected forwarding, service and security functions.

The fundamental architecture can be summarised as: Kubernetes intent → Cilium control plane → eBPF programs and maps → Linux kernel execution → validated network behaviour.

Next section: eBPF Datapath and Packet Processing — a deeper investigation of TC, XDP, socket hooks, packet redirection, map lookups, service translation and the forwarding decisions made as traffic moves between Kubernetes workloads and external networks.

Network Insight · Engineering Guide 21 · Lab 01

Cilium and eBPF Architecture Explorer

Explore how Cilium connects Kubernetes control-plane information with programmable networking inside the Linux kernel. Investigate the Cilium agent, operator, endpoint identities, eBPF programs, maps, CNI integration, and the forwarding path between Kubernetes workloads.

Kubernetes, Cilium and Linux Kernel Architecture

Select any component to inspect its role. Gold paths represent control-plane interactions; green paths represent packet forwarding.

KUBERNETES NODE · CILIUM ARCHITECTURE KUBERNETES / USER-SPACE CONTROL PLANE LINUX KERNEL / eBPF DATAPATH KUBERNETES API Cluster State CILIUM AGENT Per-Node Control CILIUM OPERATOR Cluster Coordination SOURCE POD 10.0.1.10 Identity 1001 eBPF PROGRAMS TC / Socket Hooks Policy + Forwarding eBPF MAPS Identity / Policy / CT Shared Kernel State DESTINATION POD 10.0.1.20 Simplified conceptual model · selected hooks depend on configuration
Control plane Datapath Failure
READY Cilium agent
LOADED eBPF program
READY Packet verdict
Select an architecture component to inspect its responsibilities.

Architecture Engineering Console

Architecture State Controls
Cilium agent operational
Required eBPF program attached
Identity and policy map populated
Network policy permits traffic
Destination endpoint available
Configure the architecture and execute an eBPF packet investigation.

The Cilium agent manages and reconciles datapath state. Existing kernel programs may continue forwarding traffic during a temporary agent interruption.

Cilium Control-Plane and Kernel Inventory

Component Execution Responsibility Status

Packet Processing Investigation

1. Source workload generates traffic
2. Packet reaches an applicable kernel hook
3. eBPF program performs map lookups
4. Identity-based policy is evaluated
5. Forwarding decision is applied
6. Packet reaches the destination

eBPF Architecture Diagnostic Console

[NETWORK INSIGHT · CILIUM LAB 01] Kubernetes node initialized. Cilium agent: READY eBPF datapath: LOADED Identity maps: POPULATED Endpoint: AVAILABLE Awaiting packet investigation.

Engineering Interpretation

User space: Cilium agent watches Kubernetes state and manages datapath configuration.
Kernel space: eBPF programs process packets and consult maps for relevant state.
Cluster coordination: The Cilium operator manages selected cluster-wide responsibilities.

Production Engineering Notes

Cilium typically deploys an agent on each Kubernetes node. The agent consumes Kubernetes information, manages endpoints, programs relevant eBPF maps, and attaches datapath programs at supported kernel hooks. The Cilium operator handles selected cluster-wide functions, depending on configuration and enabled features.

eBPF execution occurs inside the Linux kernel, while program loading and orchestration occur in user space. Maps provide kernel-accessible state for networking functions such as identities, connection tracking, service translation, and policy decisions.

TC, XDP, and socket hooks are different execution points. XDP operates early on supported ingress interfaces, TC can process packets at traffic-control hooks, and socket-level programs can support selected service and connection operations. They are not necessarily sequential stages of every packet's journey.

Educational simulation. The topology represents conceptual responsibilities, not a complete kernel call graph. Real Cilium behavior depends on the installed version, kernel capabilities, routing mode, enabled features, and configured network policies.

Engineering Guide 21 · Editorial Section 02

eBPF Datapath and Packet Processing

The Cilium datapath uses Linux eBPF programs to implement selected networking, security and service-handling functions at supported kernel attachment points.

Rather than treating every packet as a sequence of traditional iptables rules, Cilium can use programmable kernel hooks and shared eBPF maps to make forwarding decisions based on endpoint identity, service state, connection tracking and network policy.

Understanding this datapath is essential for diagnosing packet drops, service failures, routing problems and unexpected policy behaviour in Kubernetes.

02.1 / Kernel Datapath Architecture

Following a Packet Through the Linux Kernel

A packet entering or leaving a Kubernetes workload may encounter eBPF programs attached to supported networking hooks. The exact processing path depends on the traffic type and the configured Cilium datapath.

Socket-level service handling, TC-based packet processing and XDP-based early packet processing represent different execution opportunities rather than a mandatory sequence for every packet.

CILIUM eBPF DATAPATH — PACKET PROCESSING MODEL KUBERNETES POD Application 10.0.1.10 LINUX KERNEL / CILIUM DATAPATH Kernel Hook TC / Socket / XDP Policy Logic Identity / Rules Service / Route Backend / Forwarding eBPF Maps Endpoint Identity · Service State · Conntrack Policy State · Routing / Tunnel Metadata Network Destination Pod / Node / External PROGRAM EXECUTION AND MAP LOOKUPS
Figure 2 — Simplified conceptual packet processing architecture. Kernel hooks and policy/service decisions are shown as logical functions; real packet processing paths vary by traffic type, attachment point and Cilium configuration.
02.2 / Kernel Attachment Points

TC, XDP and Socket-Level eBPF

eBPF programs can execute at different points in the Linux networking stack. Each hook provides a different context and set of supported actions.

HOOK 01
Traffic Control (TC)

TC ingress and egress hooks can support packet processing, policy enforcement, forwarding and service handling on relevant network interfaces.

HOOK 02
Express Data Path (XDP)

XDP provides early packet processing for supported network devices and drivers, enabling selected filtering and forwarding operations before normal stack processing.

HOOK 03
Socket / cgroup Hooks

Supported socket hooks can influence connection and service handling before conventional packet forwarding occurs.

Important: TC, XDP and socket hooks are different execution contexts. A packet does not necessarily traverse all three. Engineers must identify the actual configured hook when investigating behaviour.

02.3 / eBPF Maps and State

How Kernel Programs Make Forwarding Decisions

eBPF maps provide data structures that allow programs to retrieve endpoint, service, policy and connection state.

The datapath can use map lookups to select a service backend, determine an endpoint identity or evaluate relevant forwarding and security information.

CONCEPTUAL eBPF MAP LOOKUPS PACKET: Source IP: 10.0.1.10 Destination IP: 10.96.0.50 Protocol: TCP Destination: Port 443 SERVICE LOOKUP: 10.96.0.50:443 → Backend group BACKEND SELECTION: Selected endpoint: 10.0.2.25:8443 IDENTITY LOOKUP: Source endpoint identity → Security identity POLICY EVALUATION: Identity + traffic context → Allow / Deny CONNECTION STATE: Flow tuple → Conntrack lookup/update FORWARDING: Selected backend / route → Destination path NOTE: Actual lookup order and implementation vary by Cilium datapath and feature.
02.4 / Packet Journey

Pod-to-Pod and Pod-to-Service Forwarding

Pod-to-pod traffic and pod-to-service traffic may follow different processing paths.

Direct pod connectivity generally requires endpoint routing and policy evaluation. Service traffic additionally requires a mechanism to select an appropriate backend.

01 Application Creates traffic
02 eBPF Hook Supported attachment
03 Policy Identity evaluation
04 Service / Route Backend or next hop
05 Destination Endpoint forwarding

The sequence above is a conceptual representation. In supported configurations, socket-level load balancing may occur before packet-level processing, and connection tracking can influence subsequent packets in a flow.

02.5 / Packet Redirection

Forwarding Without Traditional Per-Packet User-Space Processing

eBPF programs can use supported kernel mechanisms to redirect packets, select interfaces or influence forwarding decisions without requiring every packet to be passed to a user-space networking application.

Depending on configuration, Cilium may use native routing, encapsulation or other supported datapath mechanisms to deliver packets between nodes and endpoints.

Performance principle: eBPF can reduce certain packet-processing overheads, but performance depends on workload characteristics, kernel capabilities, map access patterns, encapsulation, policy complexity and the selected forwarding mode.

02.6 / Connection Tracking

Stateful Flow Handling in the eBPF Datapath

Connection tracking allows the datapath to maintain information about supported network flows.

This state can be used to support service translation, forwarding consistency and security enforcement. Engineers should distinguish Cilium-managed BPF connection tracking from other Linux connection-tracking mechanisms.

CONCEPTUAL CONNECTION STATE FLOW: 10.0.1.10:52000 → 10.0.2.25:8443 PROTOCOL: TCP STATE: Connection established SERVICE CONTEXT: Original service: 10.96.0.50:443 SELECTED BACKEND: 10.0.2.25:8443 RETURN TRAFFIC: Processed according to applicable translation and connection-tracking state. TROUBLESHOOTING: Inspect BPF conntrack state. Check map capacity. Validate backend availability. Confirm return-path behaviour.
02.7 / Packet Drop Investigation

Why eBPF Datapath Packets May Be Dropped

A packet drop can result from network policy, missing routing state, unsupported forwarding conditions, invalid service state or other datapath constraints.

Failure Category Possible Cause Investigation
Policy denial Traffic not permitted by applicable policy. Inspect endpoint identity and policy verdict.
Routing failure Destination or forwarding information unavailable. Inspect routes, endpoint state and datapath configuration.
Service failure Backend selection or service state problem. Inspect service and backend mappings.
Conntrack issue State exhaustion, inconsistency or timeout. Review BPF conntrack maps and relevant metrics.
MTU / encapsulation Packet size or tunnel overhead issue. Validate MTU, path MTU and forwarding mode.
02.8 / Production Diagnostics

Inspecting the Cilium Datapath

Cilium provides diagnostic capabilities for examining endpoint state, service mappings, policy behaviour, BPF maps and datapath events.

ILLUSTRATIVE CILIUM DIAGNOSTICS # Endpoint state cilium-dbg endpoint list # Service mappings cilium-dbg service list # BPF map inventory cilium-dbg bpf map list # BPF connection tracking cilium-dbg bpf ct list global # Datapath event stream cilium-dbg monitor # Drop-focused monitoring cilium-dbg monitor --type drop # Policy state cilium-dbg policy get NOTE: Run cilium-dbg in a suitable Cilium agent environment. Command availability and syntax depend on version.
✓Confirm endpoint and identity state.
✓Inspect service backend mappings.
✓Validate effective network policy.
✓Review BPF connection-tracking state.
✓Check routing and MTU configuration.
✓Correlate drops with application symptoms.
02.9 / Engineering Summary

Understanding the Programmable Kernel Datapath

The Cilium eBPF datapath provides a programmable networking architecture in which supported kernel programs and maps implement selected forwarding, security and service functions.

The essential processing model is: workload traffic → supported kernel hook → eBPF logic and state → forwarding or drop decision → destination.

Next section: Kubernetes Pod Networking and IPAM — endpoint addressing, CNI provisioning, pod routing, native versus overlay networking, VXLAN and Geneve, multi-node forwarding and production connectivity troubleshooting.

Network Insight · Engineering Guide 21 · Lab 02

eBPF Packet Journey Simulator

Follow a packet through the Linux networking stack and investigate how Cilium uses eBPF programs, kernel hooks, maps, and forwarding decisions to implement high-performance Kubernetes networking.

Linux Kernel eBPF Datapath Architecture

The visualization separates early ingress, TC processing, map lookup, and forwarding. Socket-level load balancing is modeled as a separate execution perspective.

LINUX KERNEL · eBPF PACKET PROCESSING NETWORK NIC Packet Ingress XDP HOOK Early Ingress TC eBPF PROGRAM Policy / Forwarding eBPF MAP LOOKUPS CT / Policy / Services FORWARDING Redirect / Pass DESTINATION Kubernetes Pod Conceptual packet path · actual hooks depend on configuration
eBPF execution Forwarded Dropped
TC Selected hook
READY Packet verdict
3 Map categories
Select a datapath component to inspect its role in Linux packet processing.

eBPF Packet Engineering Console

Kernel State Controls
XDP permits ingress
TC eBPF program attached
Policy map permits traffic
Connection tracking available
Service / forwarding map available
Redirect target available
Select a kernel hook and run a packet investigation.

eBPF Map and Hook Inspector

Kernel Component Function Status

Map categories are conceptual. Actual Cilium installations use multiple maps and program types whose names and availability depend on version and enabled features.

Packet Processing Pipeline

1. Packet enters selected datapath
2. Applicable eBPF hook executes
3. Kernel map lookup
4. Security policy decision
5. Forwarding / redirect
6. Destination delivery

Kernel Packet Diagnostic Console

[NETWORK INSIGHT · eBPF LAB 02] Kernel datapath initialized. TC program: ATTACHED XDP ingress: PERMIT Policy map: READY Connection tracking: READY Forwarding map: READY Awaiting packet investigation.

Engineering Interpretation

XDP: Executes early on supported ingress network interfaces.
TC: Supports packet processing at traffic-control hooks.
Socket hooks: Can implement selected service translation operations before packet forwarding.

Production Engineering Notes

XDP, TC, and socket-level eBPF hooks operate at different points in the Linux networking stack. XDP can process ingress traffic before normal socket-buffer allocation on supported drivers and interfaces. TC programs process packets through traffic-control attachment points.

Cilium uses eBPF maps to maintain networking state, including connection tracking, policy decisions, service translation, and endpoint information. These operations depend on the configured datapath and supported kernel capabilities.

Socket-level service load balancing can translate destinations at connection-related hooks, avoiding some later packet-level translation work. It should not be modeled as another mandatory stage after XDP and TC for every packet.

Educational simulation only. The packet animation is a conceptual teaching model, not an exact Linux kernel call graph or a real-time trace from a running Cilium cluster.

Engineering Guide 21 · Editorial Section 03

Kubernetes Pod Networking and IPAM

Cilium connects Kubernetes workloads through a programmable datapath while managing endpoint addressing, network interfaces, forwarding state and policy integration.

Every Kubernetes pod requires a network identity and a path to other workloads. Cilium supports multiple IP address management (IPAM) approaches and both native-routed and encapsulated networking designs.

The choice of IPAM and forwarding mode determines how pod addresses are allocated, how nodes reach remote pod networks, which routes the underlying infrastructure must understand, and how engineers troubleshoot packet delivery.

03.1 / Multi-Node Architecture

How Cilium Connects Pods Across Kubernetes Nodes

The architecture below shows two Kubernetes nodes, each with its own pod network. Select a forwarding mode to compare how traffic can move between nodes.

KUBERNETES CLUSTER — CILIUM MULTI-NODE CONNECTIVITY NODE A · 192.168.10.11 NODE B · 192.168.10.12 Frontend Pod 10.0.1.10 Other Pod 10.0.1.20 Backend Pod 10.0.2.20 Other Pod 10.0.2.30 Cilium eBPF Datapath Identity · Policy · Forwarding Cilium eBPF Datapath Identity · Policy · Forwarding Node Network Interface Underlay / Tunnel Endpoint Node Network Interface Underlay / Tunnel Endpoint Underlay IP Network Node Reachability INTER-NODE POD CONNECTIVITY Native: pod routes Overlay: node-to-node tunnel ILLUSTRATIVE POD CIDRS: NODE A 10.0.1.0/24 · NODE B 10.0.2.0/24
Figure 3 — Conceptual two-node topology. Native routing requires reachability to remote pod addresses or prefixes; overlay routing transports encapsulated traffic between reachable node tunnel endpoints. IP examples are illustrative.
Native Routing — Underlay-Aware Pod Connectivity

Traffic to a remote pod is forwarded using routes that ultimately provide reachability to the destination pod address. The underlay or node routing architecture must support the relevant pod-prefix forwarding.

03.2 / Kubernetes CNI

From Pod Creation to a Working Network Endpoint

Kubernetes delegates pod network setup to the container runtime and configured CNI implementation. Cilium's integration provisions endpoint networking and coordinates datapath state with the running Cilium agent.

01 / NETWORK NAMESPACE
Pod Network Environment

The runtime establishes the pod sandbox and its network namespace, creating the environment into which networking interfaces are configured.

02 / ADDRESS ALLOCATION
IPAM and Interfaces

The selected IPAM implementation assigns or coordinates pod addresses. Cilium configures the appropriate interface and routing integration.

03 / ENDPOINT STATE
Identity and Datapath

Cilium manages the endpoint, determines its security identity, and prepares relevant eBPF policy and forwarding state.

Engineering distinction: A pod receiving an IP address does not prove that cross-node routing, service connectivity or security-policy state is correct. Validate each layer independently.

03.3 / IP Address Management

Understanding Cilium IPAM Models

Cilium supports several IPAM strategies. The available options and their operational requirements depend on the deployment platform, Cilium version and cloud integration.

IPAM ApproachArchitectureOperational Consideration
Kubernetes PodCIDRUses pod CIDRs allocated to nodes by Kubernetes, with Cilium managing addresses from those ranges.Requires appropriate Kubernetes node CIDR allocation and routing design.
Cluster-Pool IPAMCilium operator allocates per-node pod CIDR ranges from configured cluster pools.Plan pool capacity, mask sizes and non-overlapping ranges.
Multi-Pool IPAMSupports selecting addresses from multiple configured pools in supported deployments.Requires explicit pool design and compatibility validation.
Cloud ENI / Cloud IPAMIntegrates with supported cloud networking address allocation, such as AWS ENI-based designs.Cloud quotas, subnet capacity, permissions and node interface limits matter.

For example, a cluster-pool design could reserve 10.0.0.0/16 for workload addressing and assign separate /24 pod ranges to individual nodes. These are illustrative planning values, not Cilium defaults.

ILLUSTRATIVE CLUSTER ADDRESS PLAN Cluster Pod Pool: 10.0.0.0/16 Node A Pod CIDR: 10.0.1.0/24 Node B Pod CIDR: 10.0.2.0/24 Node C Pod CIDR: 10.0.3.0/24 Node A Underlay: 192.168.10.11 Node B Underlay: 192.168.10.12 Node C Underlay: 192.168.10.13 DESIGN REQUIREMENTS: - Avoid overlapping pod ranges. - Reserve capacity for node growth. - Validate cloud subnet limitations. - Define pod-prefix reachability. - Verify service CIDR separation. - Plan multi-cluster address space.
03.4 / Native Routing

When the Physical or Cloud Network Routes Pod Traffic

In native-routing mode, Cilium does not require an encapsulation overlay for ordinary cross-node pod forwarding. Remote pod addresses must instead be reachable through the routing architecture.

Routes may be established through the underlying network, node routing mechanisms or supported route-advertisement integrations. The exact design depends on the cluster topology.

CONCEPTUAL NATIVE ROUTING SOURCE: Pod A 10.0.1.10 Node A 192.168.10.11 DESTINATION: Pod B 10.0.2.20 Node B 192.168.10.12 REQUIRED: Forwarding path to 10.0.2.0/24 CONCEPTUAL ROUTE: 10.0.2.0/24 via 192.168.10.12 The actual route may be installed on a node, router or both, depending on the native-routing architecture.

Production risk: If the underlay cannot reach the destination pod network, an otherwise healthy Cilium endpoint may still experience cross-node connectivity failures.

03.5 / Encapsulation

VXLAN and Geneve Overlay Networking

Overlay networking transports pod traffic inside an outer packet addressed between reachable tunnel endpoints, typically Kubernetes nodes. This reduces the requirement for the underlay to maintain routes to every pod prefix.

Cilium supports VXLAN and Geneve in relevant tunnelling configurations. The selected mode adds encapsulation overhead, which must be considered when determining workload MTU.

PropertyNative RoutingOverlay Routing
Pod-prefix visibilityUnderlying routing must provide pod reachability.Underlay primarily requires tunnel endpoint reachability.
EncapsulationNo Cilium overlay encapsulation for the native path.VXLAN or Geneve encapsulation.
MTUDepends on underlying path.Must account for tunnel overhead.
OperationsInvestigate pod routes and underlay forwarding.Investigate tunnel state, node reachability and MTU.
TradeoffPotentially simpler packet format, greater routing integration.Simpler underlay requirements, additional encapsulation complexity.
03.6 / Cross-Node Packet Journey

Tracing Pod A to Pod B

Consider a frontend pod on Node A communicating with a backend pod on Node B. The packet must pass the applicable policy and forwarding decisions and then traverse the inter-node network.

EXAMPLE: POD A → POD B 1. Pod A sends traffic to 10.0.2.20. 2. Packet reaches the configured Cilium-managed datapath. 3. Applicable identity-based policy is evaluated. 4. Forwarding state determines the remote-node path. 5A. NATIVE MODE: Packet uses pod-address routing without Cilium tunnel encapsulation. 5B. OVERLAY MODE: Packet is encapsulated toward Node B's tunnel endpoint. 6. Node B receives traffic. 7. Relevant ingress processing and policy are applied. 8. Packet reaches Pod B. 9. Return traffic follows the applicable reverse path.

The actual order and location of policy checks, routing lookups and connection-tracking operations depend on the enabled datapath features. This sequence is intended as an engineering investigation model.

03.7 / Production Failure Patterns

Why Pod Connectivity Breaks

SymptomLikely Investigation AreaValidation
Same-node works; cross-node failsUnderlay routing, tunnel reachability, firewall rules or MTU.Compare node-to-node and pod-to-pod paths.
Pod cannot obtain an addressIPAM pool capacity, operator, cloud quotas or CNI lifecycle.Inspect Cilium status and pod events.
Only large packets failMTU or path-MTU discovery.Test packet sizes and inspect tunnel overhead.
Selected pods cannot communicateIdentity resolution or network policy.Inspect endpoint identity and flow verdicts.
Connectivity fails after node replacementStale routes, tunnel mappings or endpoint state.Compare current node addressing with datapath state.
03.8 / Operational Diagnostics

Validating Cilium IPAM and Forwarding

Begin with endpoint readiness and assigned addresses, then identify whether the failure is in the local datapath, cross-node network or destination endpoint.

ILLUSTRATIVE DIAGNOSTICS # Kubernetes node and pod addresses kubectl get nodes -o wide kubectl get pods -A -o wide # Cilium overall status cilium status # Agent-side diagnostics cilium-dbg status cilium-dbg endpoint list cilium-dbg ip list cilium-dbg bpf tunnel list cilium-dbg monitor --type drop # Linux routing inspection ip route ip addr ip link # Kubernetes pod events kubectl describe pod POD_NAME -n NAMESPACE # Connectivity validation cilium connectivity test NOTE: Agent-side cilium-dbg commands generally run inside the Cilium agent environment. Tunnel commands apply only to supported tunnel configurations. Commands vary by version.
✓ Verify allocated pod CIDRs and address capacity.
✓ Confirm pod endpoint identity and readiness.
✓ Identify native or overlay forwarding mode.
✓ Test underlay reachability between nodes.
✓ Validate routing, tunnel state and MTU.
✓ Inspect drops and effective network policy.
03.9 / Engineering Summary

Address Allocation and Forwarding Must Be Designed Together

Cilium pod networking depends on both correct endpoint provisioning and a reliable forwarding architecture. IPAM determines which addresses workloads receive; native routing or overlay tunnelling determines how packets reach remote nodes and pods.

The essential engineering model is pod lifecycle → address allocation → endpoint state → policy enforcement → inter-node forwarding → destination delivery.

Next section: Cilium Service Load Balancing and kube-proxy Replacement — ClusterIP, NodePort, eBPF socket load balancing, backend selection, Maglev, Direct Server Return and service troubleshooting.

Network Insight · Engineering Guide 21 · Lab 03

Kubernetes Cilium Packet Path Lab

Investigate how Cilium connects Kubernetes workloads across nodes. Compare native routing with VXLAN and Geneve overlays, inspect endpoint and IPAM state, calculate encapsulation overhead, and troubleshoot inter-node connectivity failures.

Two-Node Cilium Networking Architecture

Follow traffic from Pod A on Node 1 across the underlay to Pod B on Node 2. Select any component for its engineering responsibilities.

KUBERNETES CLUSTER · INTER-NODE NETWORKING WORKER NODE 1 WORKER NODE 2 Node IP: 192.168.10.11 Node IP: 192.168.10.12 SOURCE POD A 10.1.0.10 CILIUM eBPF Routing / Encapsulation DESTINATION POD B 10.2.0.20 CILIUM eBPF Decapsulation / Delivery PHYSICAL UNDERLAY Node-to-Node Transport Simplified packet journey · encapsulation depends on routing mode
Cilium datapath Forwarding Failure
VXLAN Routing mode
1450 Effective MTU (bytes)
READY Packet verdict
Select a topology component to inspect its networking responsibilities.

Cilium Pod Networking Console

Network State Controls
Pod IP allocation successful
Destination pod CIDR reachable
Overlay transport available
Destination endpoint ready
Configure the network and execute a packet trace.

Overlay overhead assumes IPv4 outer transport and a baseline 50-byte VXLAN or Geneve encapsulation. Geneve options, IPv6 outer headers, and additional encapsulation can increase overhead.

Pod IPAM and Routing Inspector

Component Network State Status

Inter-Node Packet Processing Pipeline

1. Source pod generates packet
2. Cilium endpoint processing
3. Destination route selection
4. Optional overlay encapsulation
5. Underlay transport
6. Destination endpoint delivery

Cilium Packet Diagnostic Console

[NETWORK INSIGHT · CILIUM LAB 03] Node 1: 192.168.10.11 Node 2: 192.168.10.12 Source Pod: 10.1.0.10 Destination Pod: 10.2.0.20 Routing mode: VXLAN Underlay MTU: 1500 Effective MTU: 1450 Awaiting packet trace.

Engineering Interpretation

IPAM: Cilium assigns or manages pod addresses according to the configured IPAM mode.
Routing: Native routing relies on underlay reachability to destination pod prefixes.
Overlay: VXLAN or Geneve encapsulates pod traffic for transport between nodes.

Production Engineering Notes

Cilium can operate using native routing or encapsulation-based routing. Native routing requires the surrounding network to provide appropriate reachability to pod prefixes. Overlay routing transports pod packets between nodes using tunnel encapsulation.

VXLAN and Geneve introduce additional headers. For a baseline IPv4 outer transport, the encapsulation overhead is commonly approximately 50 bytes. Real overhead depends on outer IP version, Geneve options, and other networking features.

IP address management is deployment-dependent. Cilium supports several IPAM approaches, including Kubernetes-based, cluster-pool, and cloud-specific modes. The simulator uses representative pod addresses rather than querying a real Kubernetes cluster.

Educational simulation. This lab models selected connectivity dependencies and assumes packets are not fragmented. A real deployment may use PMTU discovery, ICMP feedback, TCP MSS adjustment, or different encapsulation and forwarding mechanisms.

Engineering Guide 21 · Editorial Section 04

Cilium Service Load Balancing and kube-proxy Replacement

Cilium can implement Kubernetes Service connectivity using eBPF-based load balancing, replacing supported kube-proxy forwarding functions and enabling additional datapath optimisations.

Traditional Kubernetes Service implementations often rely on kube-proxy programming iptables or IPVS rules. With supported kube-proxy replacement configurations, Cilium maintains Service and backend state in eBPF maps and executes relevant load-balancing logic at kernel hooks.

The architecture supports Kubernetes Service types including ClusterIP, NodePort and LoadBalancer, subject to feature support and deployment configuration.

04.1 / Service Architecture

From Virtual Service Address to Application Backend

A Kubernetes Service provides a stable logical destination while its backend endpoints may change as pods scale, restart or move between nodes.

Cilium watches relevant Kubernetes Service and endpoint resources and maintains corresponding datapath state for supported service handling.

CILIUM eBPF SERVICE LOAD BALANCING ARCHITECTURE Client Pod 10.0.1.10 ClusterIP Service 10.96.0.50:443 Logical Destination eBPF Service Maps Frontend → Backends Selection / CT State Backend A 10.0.2.11:8443 Backend B 10.0.2.12:8443 Backend C 10.0.2.13:8443 Linux Kernel eBPF Datapath Socket-Level or Packet-Level Service Handling — Configuration Dependent
Figure 4 — Logical Service selection model. The ClusterIP is a virtual destination, not necessarily a separate packet-processing hop. Service lookup may occur at socket or packet hooks depending on configuration.
04.2 / Service Types

ClusterIP, NodePort and LoadBalancer

SERVICE 01
ClusterIP

Provides a stable virtual IP for in-cluster access. Supported Cilium eBPF load balancing selects a backend for the Service.

SERVICE 02
NodePort

Exposes a Service through a port on supported node addresses. External traffic can be handled through the configured Cilium NodePort datapath.

SERVICE 03
LoadBalancer

Integrates a Kubernetes Service with an external address provisioning mechanism. Cilium handles supported datapath forwarding; external IP allocation and advertisement depend on deployment integrations.

Architecture distinction: kube-proxy replacement does not automatically provision a public load balancer or advertise Service addresses to the external network. Address allocation, routing advertisement and external traffic delivery must be designed separately.

04.3 / Packet Path Explorer

Compare Service Load-Balancing Mechanisms

Select a forwarding model to examine the different points at which Cilium can translate or direct Service traffic.

Socket-Level Service Load Balancing

For supported local socket operations, Cilium can resolve a Service to a backend at an applicable socket hook. This can avoid some later packet-level translation work. Behaviour depends on the enabled socket load-balancing features.

Application → Socket Hook → Service Lookup → Backend Socket Destination

04.4 / eBPF Service Maps

How Virtual Services Map to Real Endpoints

Cilium maintains datapath state for Service frontends and backend endpoints. A Service frontend identifies the logical destination; backend entries represent eligible application endpoints.

CONCEPTUAL SERVICE LOOKUP SERVICE FRONTEND IP: 10.96.0.50 Port: 443 Protocol: TCP BACKEND SET Backend A: 10.0.2.11:8443 Backend B: 10.0.2.12:8443 Backend C: 10.0.2.13:8443 REQUEST Client → 10.96.0.50:443 DATAPATH Service frontend lookup → Eligible backend set → Selection algorithm → Connection state / translation → Selected backend Actual eBPF map layout and lookup sequence vary by Cilium version and load-balancing configuration.

Connection-tracking and translation state help maintain consistent handling for supported flows. Backend changes must be reflected in the relevant datapath state as Kubernetes endpoint membership changes.

04.5 / Backend Selection

Random Selection and Maglev Consistent Hashing

Cilium supports configurable load-balancing algorithms in relevant datapath modes. Selection algorithms influence how new flows are distributed and how much backend reassignment occurs when membership changes.

AlgorithmPrincipleEngineering Consideration
RandomChooses among eligible backends using the configured random selection mechanism.Simple distribution; does not inherently provide consistent hashing across backend-set changes.
MaglevUses consistent hashing and lookup-table construction to select backends for supported flows.Can reduce remapping when backends change; requires compatible mode and configuration.

Operational principle: Maglev is not application-session replication. Consistent backend selection can reduce remapping, but a backend failure may still interrupt established application connections.

04.6 / SNAT and Direct Server Return

Designing the Forward and Return Traffic Paths

External Service traffic may use different translation and return-path strategies. These choices affect source-IP preservation, network routing, performance and troubleshooting.

ModeForward PathReturn PathDesign Consideration
SNAT-basedIngress node forwards to selected backend, applying relevant address translation.Return traffic follows the translation-aware path required by the configured mode.Source-address visibility and connection-tracking behaviour.
DSRIngress node directs traffic toward the selected backend while preserving the intended client-facing Service semantics.Backend can return directly to the client in supported configurations.Routing symmetry requirements differ; encapsulation or dispatch support and topology must be validated.

DSR can reduce return-path forwarding through the ingress node, but it is not universally suitable. Network topology, client reachability, packet format and supported Cilium features determine whether it can be deployed successfully.

04.7 / kube-proxy Replacement

Moving Kubernetes Service Processing into eBPF

With supported kube-proxy replacement enabled, Cilium can implement Kubernetes Service forwarding functions that would otherwise be handled by kube-proxy.

Engineers must validate the required kernel capabilities, Service features, device selection, NodePort reachability and interactions with existing networking components before removing kube-proxy from an established cluster.

CONCEPTUAL SERVICE ARCHITECTURE TRADITIONAL DESIGN Kubernetes Service state → kube-proxy → iptables / IPVS rules → Service forwarding CILIUM REPLACEMENT DESIGN Kubernetes Service state → Cilium agent → eBPF service/backend maps → Supported kernel hooks → Service forwarding VALIDATE: ClusterIP connectivity NodePort reachability LoadBalancer traffic ExternalTrafficPolicy behaviour Source-IP preservation Backend failover Host-network workloads Kernel feature compatibility
04.8 / Production Troubleshooting

Diagnosing Cilium Service Failures

SymptomInvestigation AreaValidation
ClusterIP unreachableService map state, backend eligibility, policy or routing.Compare Kubernetes Service endpoints with Cilium service state.
NodePort unavailable externallyDevice selection, firewall, routing, kube-proxy replacement configuration.Test node reachability and external forwarding path.
Intermittent backend failuresEndpoint readiness, backend churn, connection state.Correlate backend selection with pod readiness and events.
Incorrect source IPSNAT, DSR or external traffic policy.Inspect configured forwarding mode and observed source addresses.
Return traffic missingAsymmetric routing, translation state or unsupported DSR topology.Trace both forward and return paths.
ILLUSTRATIVE SERVICE DIAGNOSTICS # Kubernetes Service and endpoints kubectl get svc -A kubectl get endpointslices -A # Overall Cilium status cilium status # Agent-side diagnostics cilium-dbg status cilium-dbg service list cilium-dbg bpf lb list cilium-dbg bpf ct list global cilium-dbg monitor --type drop # Connectivity validation cilium connectivity test NOTE: Run cilium-dbg in a suitable Cilium agent environment. Command support and syntax depend on Cilium version.
✓ Verify Kubernetes Service and EndpointSlice state.
✓ Confirm Cilium service frontend and backend mappings.
✓ Check applicable network policy and endpoint readiness.
✓ Validate source-IP preservation and return routing.
✓ Confirm kube-proxy replacement and device configuration.
✓ Test backend failure and reconvergence behaviour.
04.9 / Engineering Summary

Service Connectivity Is a Datapath and Routing Problem

Cilium's eBPF service implementation connects Kubernetes Service definitions to kernel-level forwarding mechanisms. The Service frontend, backend membership, selection algorithm, translation state and return path must all operate correctly.

The central engineering model is Kubernetes Service → Cilium service state → eBPF lookup → backend selection → forwarding and return-path validation.

Next section: Cilium Network Policies and Zero Trust Security — endpoint identities, L3/L4/L7 enforcement, DNS-based policy, default-deny behaviour, policy troubleshooting and production security design.

Network Insight · Engineering Guide 21 · Lab 04

eBPF Service Load Balancer Simulator

Explore how Cilium implements Kubernetes Service connectivity using eBPF maps and load-balancing programs. Investigate ClusterIP, NodePort, socket-level translation, backend selection, Maglev-style consistent hashing, and Direct Server Return.

Cilium eBPF Service Networking Architecture

Follow a client connection through the Cilium service translation layer to a selected backend. The return path changes according to the forwarding mode.

CILIUM · KUBERNETES SERVICE DATAPATH CLIENT 10.1.0.10 TCP Connection eBPF SERVICE LB 10.96.0.100:80 Service / Backend Maps Hash + Translation BACKEND 1 10.2.0.11:8080 BACKEND 2 10.2.0.12:8080 BACKEND 3 10.2.0.13:8080 Conceptual service forwarding · DSR requires supported configuration
Service translation Selected backend Return traffic
RANDOM Backend selection
— Selected backend
READY Connection verdict
Select a component to inspect its service networking role.

Service Load Balancing Console

Service Datapath State
eBPF service map populated
Network policy permits traffic
DSR return routing available
Backend Availability
Backend 1 · 10.2.0.11
Backend 2 · 10.2.0.12
Backend 3 · 10.2.0.13
Configure service mode, algorithm, and backend availability.

eBPF Service and Backend Map Inspector

Backend Endpoint Availability

Service Connection Processing Pipeline

1. Client targets Service VIP or NodePort
2. eBPF service map lookup
3. Backend selection
4. Policy and forwarding decision
5. Backend receives traffic
6. Return path completes

eBPF Service Diagnostic Console

[NETWORK INSIGHT · CILIUM LAB 04] Service VIP: 10.96.0.100:80 Backend count: 3 Service map: READY Policy: ALLOW Awaiting service connection.

Engineering Interpretation

ClusterIP: Provides an internal virtual service address.
Maglev: Uses consistent hashing to reduce connection remapping when backend membership changes.
DSR: Allows a supported backend to return traffic directly without traversing the original load-balancing node.

Production Engineering Notes

Cilium can replace kube-proxy for supported Kubernetes Service functionality using eBPF-based load balancing. The relevant programs and maps implement Service lookup, backend selection, and packet translation.

Socket-level load balancing may perform service translation at connection-related hooks. Packet-level load balancing operates at applicable datapath hooks. The exact behavior depends on Cilium configuration, workload type, and kernel support.

Maglev provides consistent hashing intended to reduce backend remapping during membership changes. The simulator uses a simplified deterministic hash for teaching rather than reproducing Cilium's complete Maglev lookup-table construction.

Direct Server Return requires a supported deployment and compatible return routing. SNAT and DSR should be evaluated against source-IP preservation, network topology, operational complexity, and service exposure requirements.

Educational simulation only. Backend health, hashing, packet translation, and return paths are simplified. The widget does not query Kubernetes or Cilium APIs and does not change any live cluster configuration.

Engineering Guide 21 · Editorial Section 05

Cilium Network Policies and Zero Trust Security

Cilium implements workload-aware security using endpoint identities, declarative network policies and supported eBPF enforcement mechanisms.

Rather than depending exclusively on fixed IP addresses, Cilium can derive security identities from Kubernetes labels and other supported endpoint metadata. Policies can then describe which workload groups may communicate, on which protocols and ports, and under what supported application-layer conditions.

This creates a foundation for Zero Trust network segmentation in Kubernetes, where connectivity is explicitly constrained and validated rather than assumed to be safe because workloads share a cluster.

05.1 / Zero Trust Architecture

Identity-Based Enforcement Between Kubernetes Workloads

Consider a frontend application that needs to reach an API service, while an unrelated workload should not have access to the same backend.

Cilium can evaluate the source and destination security context against applicable policy. Allowed traffic proceeds through the configured datapath; denied traffic is dropped or rejected according to the relevant enforcement mechanism.

CILIUM IDENTITY-BASED ZERO TRUST ARCHITECTURE FRONTEND NAMESPACE Frontend Pod Identity: frontend BACKEND NAMESPACE API Pod Identity: backend DATA NAMESPACE Database Pod Identity: database Untrusted Workload No matching allow rule Cilium Policy Enforcement Security Identity · L3/L4 Optional L7 Proxy Integration Envoy Proxy Supported L7 Enforcement ALLOW ALLOW DENY LOGICAL POLICY RELATIONSHIPS — NOT A LITERAL PHYSICAL PACKET PATH
Figure 5 — Conceptual Zero Trust segmentation. Workload identities and applicable policy determine permitted communication. L7 enforcement can use proxy integration; not every packet traverses Envoy.
05.2 / Security Identities

Why Workload Identity Matters More Than a Fixed Pod IP

Kubernetes pods are dynamic. They are created, replaced, rescheduled and assigned new IP addresses. Security policies tied exclusively to individual addresses become difficult to maintain.

Cilium associates managed endpoints with security identities derived from relevant labels and metadata. Policies can reference those identities to express communication intent independently of an individual pod's current IP address.

IDENTITY 01
Endpoint Labels

Relevant Kubernetes labels and metadata help determine the workload security identity.

IDENTITY 02
Policy Selectors

Selectors describe source and destination workload groups that a policy applies to.

IDENTITY 03
Kernel Enforcement

Supported datapath programs use identity and policy state to enforce network access decisions.

Engineering principle: Identity-based network policy does not replace application authentication, encryption or authorisation. Zero Trust requires controls at multiple layers.

05.3 / L3 and L4 Security

Controlling Workload Reachability and Transport Ports

Layer 3 and Layer 4 policies constrain communication using workload identity, addresses, protocols and ports. For example, an API workload may accept TCP 8080 from selected frontend workloads while denying other ingress traffic once the relevant policy enforcement applies.

ILLUSTRATIVE CILIUM NETWORK POLICY apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: allow-frontend-to-api namespace: backend spec: endpointSelector: matchLabels: app: api ingress: - fromEndpoints: - matchLabels: k8s:io.kubernetes.pod.namespace: frontend app: web toPorts: - ports: - port: "8080" protocol: TCP INTENT: Select backend API endpoints. Permit selected frontend pods to connect to TCP 8080. IMPORTANT: Validate label matching, namespace selectors and effective policy in the installed Cilium version.

For selected endpoints and enforced directions, traffic that does not match an applicable allow rule is denied, subject to the complete policy configuration and Cilium's policy semantics.

05.4 / Layer 7 Enforcement

Extending Security Beyond IP Addresses and Ports

Cilium supports application-layer policy for selected protocols and configurations, including HTTP-aware controls. L7 enforcement may involve Envoy proxy integration rather than being implemented entirely by packet-level eBPF instructions.

An application-layer policy can constrain requests using supported attributes such as HTTP methods and paths, helping reduce the scope of permitted API operations.

CONCEPTUAL L7 SECURITY INTENT SOURCE: Frontend workload DESTINATION: API workload TRANSPORT: TCP 8080 ALLOWED HTTP REQUEST: GET /api/products RESTRICTED REQUEST: DELETE /api/products ENFORCEMENT: Applicable Cilium L7 policy and supported proxy integration VALIDATE: Proxy readiness Protocol compatibility TLS termination/visibility Policy verdicts Application behaviour

TLS consideration: Application-layer enforcement requires visibility into the relevant protocol content. Encryption and proxy termination design determine which HTTP attributes can be inspected.

05.5 / DNS-Aware Egress Security

Controlling External Destinations by Domain Name

Cilium supports DNS-aware egress policy in supported configurations. DNS responses can help maintain destination-IP information associated with permitted fully qualified domain names.

This can simplify policies for external services whose addresses change, but DNS-aware policy must be designed carefully because caching, shared IP addresses, DNS resolution paths and application behaviour influence the effective access boundary.

ControlPurposeEngineering Consideration
FQDN matchingPermit supported external destinations by DNS name.DNS-derived IP state and cache lifecycle.
DNS proxy visibilityObserve and enforce supported DNS request behaviour.DNS transport and resolver configuration.
Egress port restrictionsLimit permitted transport protocols and ports.Required service connectivity and exceptions.
External identity handlingApply supported policy to traffic outside the cluster.Destination classification and routing context.
05.6 / Policy Decision Explorer

Compare Common Zero Trust Traffic Scenarios

Select a scenario to explore how policy intent affects communication between Kubernetes workloads.

Allowed API Traffic

Selected frontend workloads are permitted to reach the backend API on TCP 8080 by an applicable ingress allow rule. Other policy constraints must also permit the communication.

Frontend Identity → API Policy → TCP 8080 Allowed

05.7 / Default Deny and Policy Composition

Building a Predictable Security Boundary

In Kubernetes, network-policy enforcement depends on which endpoints are selected and which traffic directions are subject to policy. Cilium supports multiple policy resources, including Kubernetes NetworkPolicy and Cilium-specific policy types.

Engineers should not assume that ordinary policy resources behave like a sequential firewall rule list. Applicable allow policies are generally composed, while explicit deny policies and supported policy tiers introduce additional semantics.

Policy ConceptMeaningRisk
Endpoint selectionDefines workloads affected by a policy.Incorrect labels leave intended workloads unprotected.
Ingress isolationRestricts incoming traffic for selected endpoints.Missing required source rules break application connectivity.
Egress isolationRestricts outgoing traffic for selected endpoints.DNS, identity providers and external APIs may become unreachable.
Policy compositionCombines applicable policy resources according to supported semantics.Assuming first-match firewall behaviour can lead to incorrect conclusions.
Explicit denyUses supported Cilium deny policy features.Must account for documented precedence and feature behaviour.

Deployment principle: Introduce restrictive policies with a validated application dependency map. Include DNS, monitoring, health checks and control-plane connectivity in the design.

05.8 / Production Troubleshooting

Investigating Policy Denials and Unexpected Access

Policy troubleshooting begins by identifying the source endpoint, destination endpoint, security identities, selected policies and observed flow verdict.

ILLUSTRATIVE POLICY DIAGNOSTICS # Kubernetes policy resources kubectl get networkpolicy -A # Cilium-specific policies kubectl get cnp -A kubectl get ccnp # Cilium status cilium status # Agent-side diagnostics cilium-dbg endpoint list cilium-dbg policy get cilium-dbg monitor --type drop # Hubble flow inspection hubble observe --verdict DROPPED # Connectivity testing cilium connectivity test NOTE: cilium-dbg runs in a suitable Cilium agent environment. Hubble CLI requires a working Hubble connection. Commands vary by version.
✓ Confirm source and destination identities.
✓ Validate policy selectors and namespaces.
✓ Inspect ingress and egress enforcement.
✓ Check DNS and L7 proxy dependencies.
✓ Correlate policy verdicts with application failures.
✓ Test both permitted and prohibited traffic.
05.9 / Engineering Summary

Zero Trust Requires Identity, Enforcement and Verification

Cilium combines workload identities, declarative policy and supported eBPF enforcement mechanisms to implement Kubernetes network segmentation. Layer 7 and DNS-aware controls extend the policy model where supported, but require additional operational consideration.

The central engineering model is workload identity → applicable policy → kernel or proxy enforcement → flow verdict → operational verification.

Next section: Hubble Observability and eBPF Network Telemetry — flow events, service maps, policy verdicts, DNS visibility, distributed network troubleshooting and production monitoring architecture.

Network Insight · Engineering Guide 21 · Lab 05

Cilium Zero Trust Policy Lab

Investigate how Cilium uses workload identities and eBPF policy enforcement to secure Kubernetes traffic. Test L3/L4 rules, L7 HTTP filtering, DNS-aware egress, and default-deny behavior across application workloads.

Kubernetes Identity-Based Security Architecture

Follow traffic from a frontend workload through Cilium policy enforcement to an internal API or external destination.

CILIUM · IDENTITY-BASED POLICY ENFORCEMENT FRONTEND POD Identity 1001 app=frontend CILIUM POLICY Identity / L3-L4 L7 Proxy / DNS ALLOW / DENY API BACKEND Identity 2001 TCP 443 EXTERNAL api.example.com DNS-Aware Egress Conceptual policy evaluation · L7 and DNS require appropriate configuration
Policy evaluation Allowed Denied
1001 Source identity
L3/L4 Policy layer
READY Traffic verdict
Select a component to inspect its Zero Trust responsibilities.

Cilium Security Policy Console

Security Enforcement Controls
Default-deny policy enabled
Frontend identity allowed
TCP 443 permitted
L7 HTTP filtering enabled
Allow HTTP /health and /api
DNS-aware external egress allowed
Configure workload identity, destination, and policy controls.

Cilium Policy and Identity Inspector

Policy Component Configured Rule Status

Zero Trust Policy Evaluation Pipeline

1. Source workload identity resolved
2. Destination policy selection
3. L3 identity rule evaluated
4. L4 port policy evaluated
5. Optional L7 or DNS controls
6. Final ALLOW / DENY verdict

Cilium Policy Diagnostic Console

[NETWORK INSIGHT · CILIUM LAB 05] Source identity: 1001 Destination: API Backend Policy enforcement: ENABLED Default deny: ENABLED Awaiting traffic evaluation.

Engineering Interpretation

Security identity: Cilium derives endpoint identities from relevant Kubernetes labels.
eBPF enforcement: L3/L4 policy decisions can be enforced in the kernel datapath.
L7 inspection: Supported application-layer filtering uses additional proxy-based processing.

Production Engineering Notes

Cilium associates workload endpoints with security identities derived from relevant labels. Identity-based policy is more resilient to changing pod IP addresses than policies relying exclusively on static addresses.

Cilium policies can express L3/L4 restrictions and supported L7 rules. L3/L4 enforcement is implemented through the eBPF datapath. L7 enforcement may require traffic redirection to a proxy, depending on the configured protocol and policy.

DNS-aware egress requires appropriate DNS rules and destination policy configuration. Allowing DNS resolution does not automatically authorize every subsequent connection to the resolved addresses.

Policy enforcement is directional. Ingress and egress rules apply to selected endpoints according to policy semantics. Default-deny behavior depends on endpoint policy selection and enforcement mode, not merely the presence of Cilium in a cluster.

Educational simulation only. The policy engine models a single simplified rule set. Real Cilium policy evaluation includes policy selection, directional enforcement, identity resolution, and potentially multiple interacting policy resources.

Engineering Guide 21 · Editorial Section 06

Hubble Observability and eBPF Network Telemetry

Hubble provides network-flow visibility, security-policy verdicts and service-level observability for supported Cilium-managed traffic.

Traditional Kubernetes troubleshooting often requires engineers to correlate pod logs, node packet captures, firewall counters and routing state. Hubble adds structured flow events derived from Cilium's datapath observability mechanisms, helping engineers identify which workloads communicated, what traffic was observed and whether policy enforcement permitted or denied it.

The Hubble architecture separates node-local event collection from optional cluster-wide aggregation and visualisation, allowing teams to investigate traffic without treating the observability system as a mandatory forwarding component.

06.1 / Hubble Architecture

From eBPF Datapath Events to Cluster-Wide Visibility

Hubble receives supported flow information from Cilium's datapath event infrastructure. Node-local Hubble capabilities expose flows for a managed node, while Hubble Relay can provide a cluster-wide view by connecting to node-local Hubble endpoints.

Hubble UI and CLI clients consume the available flow information to support investigation and service dependency analysis.

HUBBLE OBSERVABILITY — MULTI-NODE TELEMETRY ARCHITECTURE KUBERNETES NODE A Application Workloads Pods / Services Cilium eBPF Datapath Flow / Drop Events Hubble Local KUBERNETES NODE B Application Workloads Pods / Services Cilium eBPF Datapath Flow / Drop Events Hubble Local KUBERNETES NODE C Application Workloads Pods / Services Cilium eBPF Datapath Flow / Drop Events Hubble Local Hubble Relay Cluster-Wide Flow Access Hubble UI / CLI Flow and Service Analysis OBSERVABILITY PIPELINE — SEPARATE FROM THE PACKET FORWARDING PATH
Figure 6 — Conceptual Hubble architecture. Hubble Relay aggregates access to node-local flow streams; it is not necessarily a persistent event store and does not sit inline in ordinary packet forwarding.
06.2 / Flow Event Model

What a Hubble Flow Can Reveal

A Hubble flow record can include workload identities, source and destination information, transport protocol, observed traffic direction, verdict and relevant event metadata.

Depending on configuration and visibility, Hubble can also expose supported DNS, HTTP and other protocol-level observations.

ILLUSTRATIVE HUBBLE FLOW EVENT TIMESTAMP: 2026-11-20T10:00:00Z SOURCE: namespace: frontend pod: web-7d9f identity: frontend DESTINATION: namespace: backend pod: api-5c4b identity: backend PROTOCOL: TCP DESTINATION PORT: 8080 TRAFFIC DIRECTION: EGRESS VERDICT: FORWARDED OBSERVATION: Supported Cilium datapath event NOTE: This is a conceptual example, not literal Hubble CLI output. Fields vary by event type and deployment configuration.
06.3 / Observability Components

Local Hubble, Relay and User Interfaces

COMPONENT 01
Node-Local Hubble

Provides access to supported flow events associated with the Cilium-managed node, subject to buffer capacity and event availability.

COMPONENT 02
Hubble Relay

Connects to node-local Hubble endpoints to expose cluster-wide flow information to compatible clients.

COMPONENT 03
Hubble UI and CLI

Enable operators to inspect flow information, identify workload communication and explore service dependencies.

Engineering distinction: Hubble improves visibility into supported Cilium datapath events. It does not guarantee that every packet or application request is retained, nor does it replace packet captures, application tracing or comprehensive log collection.

06.4 / Flow Analysis

Policy Verdicts, DNS Queries and Application Traffic

Hubble can help engineers correlate traffic observations with policy decisions and application connectivity symptoms. Select a flow category to examine a typical investigation.

Forwarded Workload Traffic

A supported flow event indicates that traffic was forwarded at the observation point. Engineers should still validate end-to-end application delivery and return traffic.

Source Workload → Cilium Datapath → Forwarded Verdict → Destination

06.5 / Service Dependency Mapping

Understanding Which Applications Communicate

Service dependency visualisation helps reveal relationships between Kubernetes workloads and services. This is useful for application discovery, segmentation planning and identifying unexpected communication paths.

However, a dependency graph represents observed activity within the available visibility window. The absence of an observed flow does not prove that no dependency exists.

Visibility TypeEngineering ValueLimitation
Source and destination workloadsIdentify communicating namespaces and pods.Depends on available identity and event metadata.
Policy verdictsInvestigate allowed and denied traffic.Event context and enforcement semantics must be interpreted correctly.
DNS observationsInvestigate name resolution and egress dependencies.Visibility depends on DNS path and configuration.
HTTP observationsInspect supported application-layer events.Requires appropriate protocol visibility and proxy integration.
Service mapVisualise observed application relationships.Not a complete inventory of all possible dependencies.
06.6 / Hubble Metrics

Exporting Network Telemetry to Monitoring Platforms

Hubble can expose supported metrics for integration with Prometheus-based monitoring architectures. Available metrics and labels depend on enabled metric handlers and configuration.

Teams can use network-flow and policy-related metrics to detect changes in connectivity patterns, monitor drops and support operational dashboards.

CONCEPTUAL METRICS PIPELINE Cilium Datapath Events | v Hubble Metric Handlers | v Prometheus Scrape | v Metrics Storage / Queries | v Dashboards and Alerts POSSIBLE OBSERVABILITY GOALS: - Monitor policy drop trends. - Track selected flow categories. - Observe DNS activity. - Correlate network events with application incidents. - Detect telemetry gaps. METRIC NAMES AND LABELS: Depend on enabled handlers and Cilium/Hubble version.

Cardinality consideration: Highly granular labels can increase metric-series counts and monitoring cost. Production metric configuration should balance diagnostic value with storage, scrape and query overhead.

06.7 / Telemetry Limitations

Interpreting Observability Data Correctly

Hubble's event stream is not a substitute for understanding the underlying datapath. Event buffering, filtering, aggregation, feature support and observation points affect which information is available.

LimitationOperational ImpactMitigation
Finite event buffersEvents may be lost under sustained load.Monitor event-loss indicators and size buffers appropriately.
Incomplete protocol visibilitySome application details are unavailable.Validate enabled protocol and proxy features.
Aggregation gapsRelay may not reach every node-local endpoint.Check Relay connectivity and node availability.
Event interpretationA forwarded event does not prove successful application processing.Correlate with application logs, traces and response behaviour.
Retention limitationsHistorical flows may not remain accessible.Use supported external export or monitoring integrations where needed.
06.8 / Production Diagnostics

Investigating Connectivity with Hubble

Start with the affected workload pair and time window. Filter by namespace, pod, verdict or protocol, then correlate observations with policy and routing state.

ILLUSTRATIVE HUBBLE DIAGNOSTICS # Check Hubble connectivity hubble status # Observe recent flows hubble observe # Observe dropped flows hubble observe --verdict DROPPED # Filter source namespace hubble observe --from-namespace frontend # Filter destination namespace hubble observe --to-namespace backend # Observe DNS traffic hubble observe --protocol dns # Observe HTTP traffic hubble observe --protocol http # Follow live events hubble observe --follow # Cilium health cilium status NOTE: Commands require a configured Hubble CLI connection. Flags, event availability and features depend on version.
✓ Confirm Hubble and Relay connectivity.
✓ Identify affected source and destination workloads.
✓ Filter flow events by verdict and protocol.
✓ Correlate DNS and policy observations.
✓ Review telemetry buffer and event-loss indicators.
✓ Validate findings against routing and application state.
06.9 / Engineering Summary

Observability Makes Kernel Networking Investigable

Hubble exposes supported Cilium datapath events through node-local and cluster-wide interfaces, enabling engineers to investigate workload communication, policy decisions and selected application protocols.

The core operational model is eBPF event generation → node-local Hubble → optional Relay aggregation → flow analysis → incident correlation.

Next section: Cilium BGP Control Plane and External Connectivity — BGP peerings, pod CIDR advertisements, Service VIP routing, external routers, route policy, multi-node availability and production troubleshooting.

Network Insight · Engineering Guide 21 · Lab 06

Hubble Network Observability Explorer

Investigate Kubernetes networking using simulated Cilium Hubble flow telemetry. Inspect workload communications, policy verdicts, DNS activity, service dependencies, and datapath forwarding failures.

Kubernetes Service Dependency Map

Select workloads to inspect their dependencies, or select a flow record to animate its observed packet path.

HUBBLE · KUBERNETES NETWORK OBSERVABILITY FRONTEND 10.1.0.10 Namespace: web API SERVICE 10.2.0.20 Namespace: app DATABASE TCP 5432 COREDNS UDP/TCP 53 EXTERNAL API HTTPS 443 Conceptual dependency graph · flow telemetry is simulated
Forwarded Dropped Diagnostic event
0 Total flows
0 Dropped flows
0% Forwarded percentage
Select a workload to inspect its observed dependencies.

Hubble Operations Console

Generate flow telemetry to begin your network investigation.
Incident Analysis
No incident data generated yet.

Hubble observes Cilium datapath events and can expose flow verdicts, identities, protocols, and supported application-layer metadata. Available fields depend on configuration and visibility.

Live Hubble Flow Records

Select any flow to inspect its source, destination, protocol, verdict, and modeled drop reason.

Flow Source Destination Protocol Verdict

Selected Flow Inspector

Select a generated flow record to inspect its network path.

Hubble Diagnostic Console

[NETWORK INSIGHT · HUBBLE LAB 06] Observation source: simulated Cilium telemetry Flow records: 0 Awaiting telemetry generation.

Production Engineering Notes

Hubble provides observability into network flows processed by Cilium. It can expose source and destination identities, forwarding verdicts, drop reasons, and supported protocol metadata.

Hubble Relay can aggregate flow information across nodes. Hubble UI provides graphical service dependency visualization. The exact scope of visibility depends on enabled components, datapath observation points, event availability, and configured flow filters.

A dropped flow is not always caused by NetworkPolicy. Routing failures, invalid destinations, unavailable services, and other datapath conditions may also produce observable failures.

Engineers should correlate Hubble observations with Cilium endpoint state, Kubernetes Service resources, policy definitions, routing tables, and application logs.

Educational simulator. All flow records are generated locally in the browser. No live Kubernetes cluster, Cilium agent, or Hubble Relay is contacted.

Engineering Guide 21 · Editorial Section 07

Cilium BGP Control Plane and External Connectivity

Cilium's BGP Control Plane allows supported Kubernetes nodes to establish BGP sessions with external routers and advertise selected network prefixes, connecting cluster networking to enterprise and data centre routing infrastructure.

While Cilium's eBPF datapath forwards workload traffic, the BGP Control Plane provides a mechanism for distributing supported routing information beyond the Kubernetes cluster.

This separation is fundamental: BGP distributes reachability information; the Linux and Cilium datapaths forward the actual packets.

07.1 / BGP Architecture

Connecting Kubernetes Nodes to the External Routing Fabric

In a typical design, Cilium-managed Kubernetes nodes establish BGP sessions with upstream routers. The nodes advertise selected pod CIDRs or supported Service address prefixes, subject to configuration and advertisement policies.

External routers then use BGP routes to determine where traffic should be forwarded toward the Kubernetes cluster.

CILIUM BGP CONTROL PLANE — EXTERNAL ROUTING FABRIC KUBERNETES NODE A · AS 65010 Pod Network 10.0.1.0/24 Cilium BGP Control Plane Cilium eBPF Datapath Policy · Service · Forwarding KUBERNETES NODE B · AS 65010 Pod Network 10.0.2.0/24 Cilium BGP Control Plane Cilium eBPF Datapath Policy · Service · Forwarding External Router A AS 65000 External Router B AS 65000 ROUTING FABRIC BGP ROUTE EXCHANGE CONTROL PLANE ADVERTISEMENTS AND DATA PLANE FORWARDING ARE DISTINCT
Figure 7 — Conceptual BGP topology. ASN values, pod CIDRs and peer placement are illustrative. Actual Cilium BGP configuration and supported advertisement types depend on version and deployment.
07.2 / BGP Control Plane Components

Peers, Advertisements and Route Selection

COMPONENT 01
BGP Peer Configuration

Defines supported peer relationships between Cilium-managed nodes and external routers, including peer addresses and AS numbers.

COMPONENT 02
Route Advertisements

Controls which supported pod CIDRs and Service address prefixes are advertised to configured peers.

COMPONENT 03
External Routing Fabric

Receives advertisements and selects forwarding paths according to its own BGP policy and routing configuration.

Critical distinction: The Cilium BGP Control Plane advertises reachability. It is not a general-purpose replacement for the Linux routing stack or the Cilium datapath, and BGP advertisements alone do not guarantee working pod-to-pod forwarding.

07.3 / Pod CIDR Advertisements

Making Kubernetes Pod Networks Reachable

In supported native-routing architectures, pod CIDR advertisements can inform external routers about the network prefixes associated with Kubernetes nodes.

For example, Node A may advertise 10.0.1.0/24 while Node B advertises 10.0.2.0/24. The upstream fabric can then forward traffic toward the node responsible for the relevant destination prefix.

ILLUSTRATIVE BGP ROUTING TABLE EXTERNAL ROUTER — AS 65000 Prefix Next Hop -------------------------------- 10.0.1.0/24 192.168.10.11 10.0.2.0/24 192.168.10.12 NODE A: Underlay: 192.168.10.11 Pod CIDR: 10.0.1.0/24 NODE B: Underlay: 192.168.10.12 Pod CIDR: 10.0.2.0/24 FORWARDING: External router → Selected next hop → Kubernetes node → Cilium datapath → Destination pod NOTE: Actual routes and next-hop behaviour depend on the routing architecture.
07.4 / Service VIP Advertisements

Advertising Kubernetes Services to External Routers

Cilium can advertise supported Service addresses through its BGP Control Plane. This enables external routers to learn routes toward Service VIPs without relying solely on static route configuration.

Service address advertisement and Service load balancing are separate functions: BGP delivers traffic toward a node, while the configured service datapath selects and forwards to an eligible backend.

Address TypePurposeEngineering Consideration
Pod CIDRAdvertise supported pod network reachability.Native-routing design and prefix ownership.
LoadBalancer VIPAdvertise supported external Service address reachability.Service allocation, backend placement and external traffic policy.
ExternalIPAdvertise supported Service ExternalIP addresses.Address ownership and external routing design.
ClusterIPAdvertise supported ClusterIP addresses where configured and supported.Service CIDR reachability and intended exposure boundaries.

Security consideration: Advertising a Service prefix does not automatically make that Service safe for external consumption. Routing reachability must be combined with firewall policy, workload security and appropriate Service exposure controls.

07.5 / Route Advertisement Explorer

Compare BGP Routing Scenarios

Select a scenario to examine the control-plane behaviour and forwarding implications.

Pod CIDR Advertisement

A Kubernetes node advertises its supported pod prefix to an upstream BGP peer. The external router installs a route according to its policy and forwards matching traffic toward the node.

Node Pod CIDR → BGP Advertisement → External RIB/FIB → Node Forwarding

07.6 / ECMP and High Availability

Multiple Next Hops for Service Reachability

When supported Service prefixes are advertised from multiple nodes, an upstream routing fabric may install multiple equal-cost next hops, subject to BGP policy and router capabilities.

ECMP can distribute traffic across eligible next hops, but the resulting traffic behaviour depends on hashing, backend availability, Service external traffic policy and the configured Cilium load-balancing mode.

CONCEPTUAL SERVICE VIP ECMP SERVICE VIP: 203.0.113.50/32 ROUTER NEXT HOPS: 192.168.10.11 192.168.10.12 TRAFFIC: Client → Service VIP → Router ECMP decision → Selected Kubernetes node → Cilium Service handling → Application backend VALIDATE: Advertised route eligibility Backend availability ExternalTrafficPolicy ECMP hashing behaviour Return-path routing Route withdrawal timing
07.7 / Route Withdrawal and Failure Recovery

What Happens When a Node or BGP Session Fails?

BGP failure recovery depends on how the failure is detected, when routes are withdrawn or invalidated, and how quickly external routers update their forwarding tables.

A failed node, lost BGP session or changed Service eligibility may affect the available routes. However, BGP session state and application health are not the same thing, and not every application failure automatically causes a route withdrawal.

FailurePotential Routing ImpactInvestigation
BGP peer failureRoutes learned from the affected session may be withdrawn after failure detection.Inspect peer state, timers and route tables.
Kubernetes node failureNode-originated advertisements may become unavailable.Check node health and upstream route convergence.
Service backend failureService forwarding eligibility changes; route behaviour depends on advertisement policy.Compare Service endpoints with advertised routes.
Underlay failureBGP may remain established or fail depending on topology.Validate next-hop reachability and forwarding path.
07.8 / Production Troubleshooting

Investigating Missing BGP Routes and Broken External Connectivity

Begin with the BGP session state and configured advertisements. Then validate whether the external router received, accepted and installed the expected route.

ILLUSTRATIVE CILIUM BGP DIAGNOSTICS # Cilium status cilium status # Kubernetes nodes kubectl get nodes -o wide # BGP custom resources kubectl get ciliumbgpclusterconfigs kubectl get ciliumbgppeerconfigs kubectl get ciliumbgpadvertisements # BGP peer status cilium bgp peers # BGP routes cilium bgp routes # Linux forwarding ip route # Service and endpoint state kubectl get svc -A kubectl get endpointslices -A NOTE: BGP resource names and CLI commands depend on Cilium version and BGP API generation. Validate supported syntax against the installed release.
✓ Confirm BGP peer establishment.
✓ Validate local and peer AS numbers.
✓ Inspect configured advertisement selectors.
✓ Compare advertised, received and installed routes.
✓ Validate next-hop reachability and ECMP.
✓ Test Service forwarding and return traffic.
07.9 / Engineering Summary

BGP Connects Kubernetes Reachability to the Network Fabric

Cilium's BGP Control Plane provides a mechanism for advertising supported pod and Service prefixes to external routing infrastructure. External routers determine how those advertisements influence forwarding, while the Cilium datapath handles relevant workload and Service traffic.

The essential engineering model is prefix ownership → BGP advertisement → external route selection → next-hop forwarding → Cilium datapath → destination workload.

Next section: Cluster Mesh, Encryption and High Availability — multi-cluster connectivity, shared identities, service discovery, WireGuard/IPsec encryption, failure domains and production resilience engineering.

Network Insight · Engineering Guide 21 · Lab 07

Cilium BGP Routing Simulator

Explore how Cilium's BGP control plane exchanges routing information with external network devices. Investigate peer establishment, prefix advertisement, service reachability, and route withdrawal.

Kubernetes BGP Control Plane Architecture

Two Cilium nodes establish eBGP sessions with an external leaf router. Each node advertises its configured prefixes. The router installs accepted routes into its routing table.

CILIUM BGP CONTROL PLANE · EXTERNAL ROUTING CILIUM NODE 1 AS 65001 192.168.10.11 10.1.0.0/24 CILIUM NODE 2 AS 65001 192.168.10.12 10.2.0.0/24 LEAF ROUTER AS 65010 BGP RIB / FIB External Network CLIENT External Network BGP distributes reachability; eBPF handles workload forwarding
Established peer Failed peer Route exchange
0/2 Established peers
0 Installed routes
READY Routing verdict
Select a topology component to inspect its BGP networking responsibilities.

BGP Routing Engineering Console

BGP Control Plane State
Node 1 BGP transport available
Node 2 BGP transport available
Pod CIDR advertisements enabled
Service VIP advertisement enabled
External router accepts prefixes
Configure BGP peering and advertisement controls, then establish the sessions.

External Router BGP Routing Table

Prefix Next Hop Origin Status

Routes appear after successful BGP advertisement and acceptance.

BGP Session and Route Processing

1. TCP/179 transport established
2. BGP OPEN and ASN validation
3. BGP session reaches Established
4. Prefix advertisement generated
5. External routing policy evaluated
6. Accepted route installed

BGP Diagnostic Console

[NETWORK INSIGHT · CILIUM LAB 07] Local AS: 65001 External AS: 65010 Peers configured: 2 Awaiting BGP session establishment.

Engineering Interpretation

BGP control plane: Advertises configured reachability information to external peers.
eBPF datapath: Processes workload forwarding independently of BGP route exchange.
Service advertisement: Requires appropriate Cilium BGP configuration and eligible Service address allocation.

Production Engineering Notes

Cilium's BGP control plane enables supported Kubernetes networking prefixes to be advertised to external BGP routers. Configuration determines which peers, address families, pod CIDRs, and Service addresses are advertised.

BGP peering does not automatically guarantee end-to-end workload connectivity. External route acceptance, next-hop reachability, Cilium datapath state, Service forwarding, and network policy must also be considered.

A BGP session may be Established while a required prefix is absent because advertisement configuration, address eligibility, or external routing policy prevents the route from being installed.

In production, engineers should inspect Cilium BGP peer status, advertised routes, external router RIB/FIB state, Kubernetes Service configuration, and datapath reachability before concluding that a BGP control-plane problem is the sole cause of an outage.

Educational simulator only. The lab models simplified eBGP peering and route advertisement. It does not implement the complete BGP finite state machine, route selection algorithm, or live Cilium configuration.

Engineering Guide 21 · Editorial Section 08

Cluster Mesh, Encryption and High Availability

Cilium Cluster Mesh extends supported workload connectivity and security identity capabilities across multiple Kubernetes clusters, enabling multi-cluster networking, global service discovery and policy-aware communication.

In production, multi-cluster architecture introduces additional failure domains, network paths and security boundaries. Engineers must design cluster connectivity, address allocation, service availability, encryption and operational recovery as a coordinated system.

Cilium also supports transparent node-to-node traffic encryption using supported WireGuard or IPsec configurations, providing protection for relevant workload traffic in transit.

08.1 / Multi-Cluster Architecture

Connecting Kubernetes Clusters Across Regions and Networks

Cluster Mesh allows supported Cilium-managed clusters to exchange the metadata and connectivity information needed for cross-cluster workload communication.

Each cluster retains its own Kubernetes control plane and local networking infrastructure. Cluster Mesh adds mechanisms for exchanging relevant identities, endpoint information and global service state.

CILIUM CLUSTER MESH — MULTI-CLUSTER NETWORK ARCHITECTURE CLUSTER A · REGION 1 Frontend Pod 10.10.1.10 API Pod 10.10.1.20 Cilium Agent / Datapath Identity · Policy · Forwarding CLUSTER B · REGION 2 API Pod 10.20.1.20 Database Pod 10.20.1.30 Cilium Agent / Datapath Identity · Policy · Forwarding Cluster Mesh Control Plane Identity / Endpoint / Service State Inter-Cluster Network Optional Encryption DATA CONNECTIVITY AND CLUSTER MESH STATE EXCHANGE ARE DISTINCT FUNCTIONS
Figure 8 — Conceptual Cluster Mesh design. Cross-cluster routing, security identities, global service configuration and encryption require supported deployment settings and network reachability.
08.2 / Cluster Mesh Components

Identity, Endpoint Discovery and Global Service State

COMPONENT 01
Cluster Identity

Participating clusters require appropriate cluster identifiers and compatible identity allocation arrangements to support cross-cluster security semantics.

COMPONENT 02
Shared State Exchange

Cluster Mesh exchanges relevant endpoint, identity and service information through its supported control-plane mechanisms.

COMPONENT 03
Global Services

Supported global service configuration can make eligible backends from multiple clusters available through a coordinated service abstraction.

Design requirement: Pod address ranges must be planned to avoid unsupported overlap. Cross-cluster datapath reachability is a separate requirement from successful Cluster Mesh control-plane connectivity.

08.3 / Global Services

Distributing Application Connectivity Across Clusters

Cilium global services can support multi-cluster service discovery and backend selection in compatible configurations. This allows selected services to expose eligible backends across participating clusters.

Engineers must distinguish between a global service definition, the current set of healthy or eligible backends, and the routing path used to reach those backends.

CapabilityPurposeEngineering Consideration
Global service discoveryExpose supported service backends across clusters.Matching service configuration and control-plane health.
Backend selectionChoose among eligible local and remote endpoints.Traffic distribution, latency and failure behaviour.
Service affinityPrefer local or remote backends where supported and configured.Capacity and resilience tradeoffs.
Cross-cluster reachabilityDeliver traffic to selected remote endpoints.Routing, firewall policy, MTU and encryption.
08.4 / Encryption Architecture

WireGuard and IPsec for Workload Traffic Protection

Cilium supports transparent encryption using supported WireGuard and IPsec datapath configurations. Encryption protects relevant traffic in transit between participating nodes, subject to feature and deployment limitations.

Encryption design must account for key distribution, kernel support, packet overhead, routing compatibility and operational recovery.

TechnologyArchitectureOperational Consideration
WireGuardUses WireGuard-based encrypted tunnels in supported Cilium configurations.Kernel compatibility, peer state, keys and MTU.
IPsecUses supported IPsec-based encryption for relevant node-to-node traffic.Key management, encryption overhead and compatibility.
Application TLSEncrypts application communication at a higher layer.Certificates, endpoint identity and application termination.

Security principle: Network-layer encryption does not automatically provide application-level authentication or authorisation. Use workload identity, network policy and application security controls together.

08.5 / Multi-Cluster Scenario Explorer

Compare Connectivity and Failure Conditions

Select a scenario to examine the expected architectural behaviour.

Healthy Multi-Cluster Connectivity

Participating clusters exchange supported Cluster Mesh state and have working network reachability. Eligible remote service backends can receive traffic according to the configured global service behaviour.

Cluster Mesh State → Backend Discovery → Cross-Cluster Forwarding

08.6 / High Availability Design

Designing for Independent Cluster and Network Failures

Multi-cluster deployment can improve resilience, but it also introduces additional dependencies. Cluster Mesh control-plane availability, inter-cluster routing, encryption state and application backend health must be evaluated separately.

CONCEPTUAL MULTI-CLUSTER HA DESIGN CLUSTER A: Region 1 Local API Backends Local Cilium Datapath CLUSTER B: Region 2 Remote API Backends Local Cilium Datapath SHARED REQUIREMENTS: Cluster Mesh connectivity Compatible security identities Supported global service state Cross-cluster routing Policy enforcement Optional encrypted transport FAILURE DOMAINS: Individual pod Kubernetes node Cilium agent Cluster Mesh control plane Inter-cluster network Regional infrastructure Application dependency OBJECTIVE: Preserve local service where possible and recover remote connectivity predictably.
08.7 / Failure Modes

Understanding Control-Plane and Data-Plane Separation

A loss of Cluster Mesh state exchange does not necessarily mean that every established cross-cluster packet flow stops immediately. Conversely, a healthy Cluster Mesh control plane does not prove that remote workload traffic is deliverable.

FailurePotential ImpactInvestigation
Cluster Mesh state exchange failureRemote identity, endpoint or service state may become stale or unavailable.Check Cluster Mesh status and control-plane connectivity.
Inter-cluster network failureRemote workloads become unreachable despite valid service metadata.Trace underlay routes and firewall policy.
Encryption peer failureProtected traffic may fail between affected nodes.Inspect encryption state, keys and peer reachability.
Remote backend failureEligible backend set changes; service continuity depends on remaining capacity and configuration.Compare service endpoints and observed traffic.
Address overlapAmbiguous routing or unsupported cross-cluster reachability.Audit pod and service address plans.
08.8 / Production Troubleshooting

Validating Cluster Mesh and Encrypted Connectivity

Start with cluster identities and Cluster Mesh health. Then validate remote endpoint discovery, inter-cluster routes, policy and encryption state.

ILLUSTRATIVE CLUSTER MESH DIAGNOSTICS # Cilium overall health cilium status # Cluster Mesh status cilium clustermesh status # Kubernetes nodes kubectl get nodes -o wide # Service endpoints kubectl get svc -A kubectl get endpointslices -A # Agent-side diagnostics cilium-dbg status cilium-dbg endpoint list # Node routing ip route ip link # Connectivity tests cilium connectivity test NOTE: Cluster Mesh commands require an appropriately configured Cilium CLI and context. Encryption diagnostics and available commands vary by Cilium version and mode.
✓ Confirm unique cluster identifiers.
✓ Validate non-overlapping pod address ranges.
✓ Inspect Cluster Mesh state exchange.
✓ Test inter-cluster routing and MTU.
✓ Validate encryption peers and key state.
✓ Test local and remote backend failure scenarios.
08.9 / Engineering Summary

Multi-Cluster Resilience Depends on More Than Connectivity

Cilium Cluster Mesh can extend workload identity, service discovery and supported networking capabilities across Kubernetes clusters. WireGuard and IPsec provide additional transport security options, while high availability requires deliberate handling of control-plane, datapath and application failures.

The essential engineering model is cluster identity → shared state → remote endpoint discovery → routed and secured forwarding → failure recovery.

Next section: Production Troubleshooting and Performance Engineering — packet drops, MTU failures, route selection, eBPF verifier issues, map pressure, connection tracking, service failures and systematic incident investigation.

Network Insight · Engineering Guide 21 · Lab 08

Cilium Multi-Cluster Failure Investigation Lab

Investigate cross-cluster Kubernetes networking using Cilium Cluster Mesh. Examine global service connectivity, remote identity synchronization, encryption transport, cluster availability, and failover behavior.

Multi-Cluster Cilium Architecture

Two Kubernetes clusters share selected service and identity information through Cluster Mesh. Application traffic traverses a separate inter-cluster network.

CILIUM CLUSTER MESH · MULTI-CLUSTER NETWORKING CLUSTER A CLUSTER B CLUSTER A Mesh Control Identity / Service CLUSTER B Mesh Control Identity / Service CLIENT POD 10.1.0.10 GLOBAL BACKEND 10.2.0.20 Control Plane Encrypted Data Path Control-plane synchronization and data-plane traffic are separate
Mesh control plane Inter-cluster traffic Failure
READY Mesh state
2 Eligible backends
READY Traffic verdict
Select a component to inspect its multi-cluster responsibilities.

Cluster Mesh Engineering Console

Cluster Health and Control Plane
Cluster Mesh synchronization healthy
Remote security identities synchronized
Cluster B available
Inter-cluster routing reachable
Encryption transport healthy
Global service synchronization healthy
Local Cluster A backend healthy
Configure the Cluster Mesh environment and execute a multi-cluster request.

Cluster Mesh State Inspector

Component State Status

Multi-Cluster Request Processing

1. Global service backend selected
2. Remote endpoint discovered
3. Remote identity evaluated
4. Inter-cluster routing validated
5. Encryption transport validated
6. Backend receives request

Cluster Mesh Diagnostic Console

[NETWORK INSIGHT · CILIUM LAB 08] Cluster A: AVAILABLE Cluster B: AVAILABLE Cluster Mesh: CONNECTED Encryption: WireGuard Awaiting multi-cluster request.

Engineering Interpretation

Cluster Mesh: Shares selected endpoint identity and service information between participating clusters.
Global services: Can expose eligible endpoints across clusters according to configured service behavior.
Encryption: WireGuard or IPsec can protect supported node-to-node traffic when correctly configured.

Production Engineering Notes

Cilium Cluster Mesh enables multi-cluster connectivity and supported cross-cluster service discovery. Cluster identities, endpoint information, and global service configuration influence how workloads communicate.

The Cluster Mesh control plane must be distinguished from the data plane. Loss of synchronization can prevent new remote endpoint or identity updates, while existing forwarding state may continue temporarily depending on the failure and deployed configuration.

Cross-cluster encryption requires compatible node connectivity, key material, and Cilium configuration. WireGuard and IPsec are alternative encryption mechanisms, not simultaneous requirements.

Global service failover depends on endpoint health visibility, service annotations or other configuration, cluster reachability, and the selected traffic distribution behavior.

Educational simulation only. The model simplifies identity synchronization, service failover, and encryption dependencies. It does not represent a complete Cluster Mesh implementation or modify any live Kubernetes cluster.

Engineering Guide 21 · Editorial Section 09

Cilium Production Troubleshooting and Performance Engineering

Reliable Cilium operations require engineers to investigate Kubernetes state, eBPF datapath behaviour, network routing, policy enforcement, service forwarding and Linux kernel resources as one connected system.

Production incidents rarely present themselves as a single obvious eBPF failure. A broken application request may originate from an IPAM problem, an incorrect security identity, a missing Service backend, an unreachable pod CIDR, an MTU mismatch or exhausted connection-tracking resources.

The objective is to develop a repeatable investigation process that moves from the observed application symptom to the exact failing component.

09.1 / Incident Investigation Architecture

Following a Failed Packet Through the Cilium Stack

When a workload cannot communicate, the most effective approach is to reconstruct the intended packet path and validate each forwarding decision.

The architecture below shows the major components involved in a typical Cilium incident investigation.

CILIUM PRODUCTION INCIDENT — END-TO-END PACKET INVESTIGATION Application Pod Source Workload Cilium Endpoint Identity / Interface eBPF Policy Allow / Deny Service / Routing Backend / Next Hop Destination Pod Application Backend Potential Failure Domains Policy · CT · MTU · Route · Map Pressure Hubble / Cilium Diagnostics Flow Verdicts · Drops · Endpoint State DIAGNOSTIC MODEL — VALIDATE CONTROL PLANE, DATA PLANE AND APPLICATION STATE
Figure 9 — Conceptual incident investigation. The diagram illustrates diagnostic relationships rather than the exact execution order of all Cilium datapath hooks.
09.2 / Failure Domains

Classifying the Problem Before Changing Configuration

DOMAIN 01
Control Plane

Cilium agent health, Kubernetes resource synchronisation, IPAM allocation, endpoint identities and BGP state.

DOMAIN 02
Data Plane

eBPF program attachment, map lookups, packet forwarding, connection tracking, policy decisions and routing.

DOMAIN 03
Infrastructure

Underlay reachability, MTU, cloud firewall rules, kernel support, network interfaces and resource pressure.

Operational rule: Do not restart Cilium agents or modify cluster-wide network configuration before collecting the evidence needed to distinguish a local endpoint issue from a systemic datapath failure.

09.3 / eBPF Program Failures

Verifier Rejections, Attachment Errors and Kernel Compatibility

eBPF programs must pass kernel verification before they can be loaded. The verifier checks properties such as memory safety and valid program behaviour. A rejected program may prevent the intended datapath feature from becoming active.

Other failures may arise from missing kernel capabilities, unsupported helpers, incompatible program types or errors attaching programs to the required hooks.

Failure TypePotential SymptomInvestigation
Verifier rejectioneBPF program cannot load.Inspect verifier output and agent logs.
Kernel capability mismatchRequired datapath feature unavailable.Validate kernel and Cilium compatibility.
Program attachment failureTraffic not processed at the intended hook.Inspect loaded programs and attachment points.
Map creation failureDatapath state unavailable or incomplete.Inspect resource limits and agent logs.
ILLUSTRATIVE eBPF DIAGNOSTICS # Kernel version uname -r # Loaded eBPF programs bpftool prog show # eBPF maps bpftool map show # Cilium agent status cilium-dbg status # Cilium agent logs kubectl -n kube-system logs \ -l k8s-app=cilium \ --tail=200 NOTE: bpftool generally requires appropriate host privileges. Verify program attachment and feature compatibility against the installed release.
09.4 / Connection Tracking and Map Pressure

When Kernel State Becomes a Performance Bottleneck

Cilium relies on supported eBPF maps for connection tracking, service state, endpoint information and other datapath functions. High connection churn or insufficient resource sizing can create pressure on relevant state tables.

Symptoms may include failed new connections, increased drop events or inconsistent behaviour under load.

CONCEPTUAL CONNECTION TRACKING CLIENT: 10.0.1.10:51000 SERVICE: 10.96.0.50:443 BACKEND: 10.0.2.20:8443 FLOW STATE MAY INCLUDE: Source / destination tuple Protocol Translation context Connection tracking metadata Relevant timeouts POTENTIAL PRESSURE: High connection creation rate Large concurrent flow count Insufficient map capacity Excessive stale state Resource contention INVESTIGATE: CT map occupancy Map allocation failures Drop reasons Connection rate CPU and memory pressure

Increasing map sizes without understanding workload behaviour can consume additional kernel memory and may not resolve the underlying problem. Capacity planning should use observed connection rates, concurrent flow counts and resource utilisation.

09.5 / Incident Scenario Explorer

Diagnose Common Production Failures

Select a failure scenario to review the likely investigation path.

Policy Drop Investigation

A workload cannot reach its destination because applicable Cilium policy does not permit the flow. Inspect endpoint identities, policy selectors, ingress and egress enforcement, and Hubble drop verdicts.

Endpoint Identity → Policy Evaluation → Drop Event → Rule Correction

09.6 / MTU and Routing

Why Small Packets Work but Large Packets Fail

MTU problems are common in overlay and encrypted networking because encapsulation adds packet overhead. A workload may successfully exchange small packets while larger packets fail or experience fragmentation-related issues.

Routing failures can produce similar symptoms but require a different investigation. Engineers must determine whether packets are being dropped because of size, missing routes, unreachable next hops or policy.

SymptomPossible CauseValidation
Small requests work, large requests failMTU or path-MTU discovery issue.Test packet sizes and inspect tunnel overhead.
Same-node works, cross-node failsUnderlay route, tunnel or firewall problem.Compare node and pod connectivity.
Only external traffic failsBGP, NAT, Service or upstream routing issue.Trace forward and return paths.
Intermittent cross-cluster failureRemote route, encryption or Cluster Mesh dependency.Validate each cluster and inter-cluster path.
09.7 / Performance Engineering

Optimising Throughput, Latency and Datapath Efficiency

eBPF can reduce some networking overhead by processing traffic at supported kernel hooks, but performance depends on workload patterns, enabled features, kernel implementation and hardware capabilities.

Optimisation requires measurement rather than assuming that every eBPF feature improves every workload.

AreaPotential OptimisationTradeoff
Service load balancingUse supported efficient backend selection and socket handling.Feature compatibility and workload behaviour.
Routing modeEvaluate native routing versus encapsulation.Underlay complexity and MTU overhead.
Connection trackingSize maps using measured flow demand.Kernel memory consumption.
Policy enforcementMinimise unnecessary complexity and validate proxy requirements.Security objectives must remain intact.
ObservabilityChoose appropriate event and metric collection levels.Diagnostic coverage versus resource overhead.

Performance principle: Benchmark representative traffic patterns before and after a configuration change. Measure latency, throughput, packet loss, CPU utilisation and application-level behaviour rather than relying on synthetic throughput alone.

09.8 / Production Diagnostic Workflow

A Repeatable Cilium Incident Investigation Process

Use a layered investigation sequence to identify the failing component and minimise unnecessary changes.

CILIUM PRODUCTION DIAGNOSTIC WORKFLOW 1. IDENTIFY THE SYMPTOM Which workloads are affected? Same-node or cross-node? Internal or external traffic? 2. VERIFY KUBERNETES STATE kubectl get pods -A -o wide kubectl get nodes -o wide kubectl get svc -A kubectl get endpointslices -A 3. CHECK CILIUM HEALTH cilium status cilium-dbg status cilium-dbg endpoint list 4. INSPECT POLICY / DROPS cilium-dbg monitor --type drop hubble observe --verdict DROPPED 5. INSPECT SERVICE STATE cilium-dbg service list cilium-dbg bpf lb list 6. INSPECT ROUTING ip route ip link ip addr 7. INSPECT KERNEL STATE bpftool prog show bpftool map show 8. TEST CONNECTIVITY cilium connectivity test 9. VALIDATE THE FIX Repeat affected traffic. Confirm expected verdicts. Check return-path delivery. Monitor recurrence. NOTE: Commands may require agent, host or Hubble access and vary by installed version.
✓ Capture the failure before making changes.
✓ Establish the affected traffic scope.
✓ Validate endpoint and Service state.
✓ Inspect eBPF maps and drop reasons.
✓ Confirm routing, MTU and return paths.
✓ Verify recovery with repeatable tests.
09.9 / Final Engineering Summary

Understanding Cilium as a Complete Networking System

Cilium is more than a Kubernetes CNI plugin. It combines programmable eBPF datapath processing, workload identity, network policy, Service load balancing, routing integrations, observability and supported multi-cluster capabilities.

Reliable production operations require engineers to understand how these components interact and how failures propagate between them.

The complete engineering model is Kubernetes intent → Cilium control plane → endpoint and eBPF state → packet forwarding → policy enforcement → telemetry → operational verification.

Guide 21 complete: You have covered Cilium architecture fundamentals, eBPF datapath processing, Kubernetes pod networking and IPAM, Service load balancing, Zero Trust policies, Hubble observability, BGP connectivity, Cluster Mesh, encryption and production troubleshooting.

Network Insight · Engineering Guide 21 · Lab 09

Cilium Network Incident Investigation Engine

Investigate production-style Kubernetes network failures using Cilium datapath state, eBPF maps, policy verdicts, routing information, DNS diagnostics, MTU analysis, and simulated Hubble observability. Identify the fault, apply a targeted repair, and verify recovery.

Cilium Production Datapath Architecture

Follow traffic through a Kubernetes application path. Each diagnostic layer can independently affect end-to-end connectivity.

CILIUM · PRODUCTION INCIDENT DATAPATH CLIENT POD 10.1.0.10 Endpoint Identity eBPF DATAPATH Policy / Service Maps / Forwarding Cilium Agent BACKEND 10.2.0.20 TCP 443 ROUTE External Network OBSERVABILITY LAYER Hubble Flows · Drop Reasons · Cilium Status · eBPF Maps Conceptual packet path · failure classification is simulated
Healthy forwarding Failure location Datapath processing
100% Health score
0 Active faults
1450 Path MTU
READY Traffic verdict
Select a component to inspect its production troubleshooting role.

Incident Response Console

Diagnostic Commands
Load an incident and begin your production investigation.

Network State Comparison Engine

Subsystem Baseline Current State

Incident Processing Pipeline

1. Endpoint health validation
2. eBPF Service map lookup
3. Security policy evaluation
4. DNS resolution
5. Routing and next-hop validation
6. MTU and packet delivery

Production Diagnostic Terminal

[NETWORK INSIGHT · CILIUM LAB 09] Cluster: production-simulated Cilium agent: RUNNING Endpoints: HEALTHY Service maps: POPULATED Network policy: ALLOW Awaiting incident investigation.

Root Cause and Recovery Analysis

Investigation workflow: Observe the failure, inspect the relevant subsystem, isolate the root cause, apply a repair, and verify connectivity.
State comparison: Compare the current network state against a healthy baseline.

Production Engineering Notes

Cilium production troubleshooting requires separating endpoint health, eBPF service translation, network policy, routing, DNS resolution, and MTU behavior. A successful control-plane operation does not guarantee that packets traverse the datapath successfully.

Hubble can provide useful flow-level evidence, including policy verdicts and supported drop reasons. However, engineers should correlate telemetry with Cilium endpoint state, Service maps, Kubernetes resources, kernel routes, and underlying network connectivity.

MTU failures are especially important in overlay environments. Encapsulation consumes additional packet overhead, and oversized packets may fail if fragmentation or Path MTU Discovery cannot resolve the mismatch.

The correct operational response is to identify the failing subsystem before applying a targeted change. Restarting components or broadly disabling security controls can conceal the underlying problem.

Educational incident simulator. Diagnostic outputs are illustrative, not live command results. The network state, repairs, and packet tests operate entirely inside this browser widget.

☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Matt Conran
Tags: No tags

Comments are closed.