Azure Kubernetes Networking Architecture

Azure Kubernetes Service Networking Architecture Explained

Network Insight · The Visual Age
ENGINEERING GUIDE 22 / 1000
Cloud Networking · Kubernetes · Azure Infrastructure

Azure Kubernetes Service Networking Architecture Explained

Understand how AKS connects Kubernetes workloads to Azure infrastructure through Azure CNI Overlay, Cilium eBPF, virtual networks, Services, load balancers, private networking and enterprise security controls.

Follow traffic from a Kubernetes pod through the node datapath and Azure network fabric. Explore how packets are routed, translated, filtered and delivered to applications in production.

Azure CNI Overlay Cilium eBPF Azure VNets AKS Services Private AKS Azure Firewall Network Policies
Explore AKS Architecture
AZURE KUBERNETES SERVICE — NETWORK ARCHITECTURE AKS Managed Control Plane API Server · Scheduling · Cluster State AZURE VIRTUAL NETWORK / AKS NODE SUBNET AKS Worker Node A Pod A Overlay IP Pod B Overlay IP Cilium / CNI Datapath AKS Worker Node B Pod C Overlay IP Pod D Overlay IP Cilium / CNI Datapath Azure Load Balancer Ingress / Service VIP Azure Routing VNet / UDR / Egress Azure Firewall Security / Egress
Conceptual architecture. Azure CNI networking mode, Cilium datapath integration, routing and firewall paths depend on the deployed AKS configuration. Control-plane communication is distinct from workload forwarding.
Architecture Foundations

Three Layers of AKS Networking

Production AKS networking spans Azure infrastructure, Kubernetes networking abstractions and the node datapath.

LAYER 01 / AZURE INFRASTRUCTURE
VNet, Subnets and Routing

Azure VNets, node subnets, NSGs, user-defined routes, load balancers, private connectivity and network security boundaries.

LAYER 02 / KUBERNETES
Pods, Services and DNS

Kubernetes pod addressing, service discovery, endpoints, ingress and cluster-level network configuration.

LAYER 03 / NODE DATAPATH
Azure CNI and Cilium

Container networking, packet processing, supported eBPF service handling, network policy enforcement and forwarding decisions.

Interactive Architecture Explorer

Follow an AKS Network Traffic Path

Select an engineering scenario to understand how Azure infrastructure and Kubernetes networking components interact.

Pod-to-Pod Communication

Traffic between Kubernetes pods is forwarded through the configured node networking datapath. Same-node communication can remain local, while cross-node communication depends on the selected AKS networking mode and node reachability.

Source Pod → CNI / Cilium Datapath → Node or Cross-Node Network → Destination Pod
Engineering Objectives

What This Guide Will Teach You

01 / ADDRESSING
Design AKS IP Architecture

Compare Azure CNI Overlay and Pod Subnet approaches, address allocation, subnet capacity and connectivity constraints.

02 / FORWARDING
Understand Packet Paths

Investigate pod-to-pod, Service, ingress, egress and hybrid network forwarding.

03 / SECURITY
Apply Network Segmentation

Combine supported Cilium network policies, Azure NSGs, firewall controls and private access.

04 / OPERATIONS
Troubleshoot Production AKS

Diagnose DNS, routing, NAT, load balancing, policy, MTU and datapath problems.

05 / RELIABILITY
Engineer Resilience

Plan availability zones, node pools, application redundancy and network failure recovery.

06 / OBSERVABILITY
Investigate eBPF Networking

Understand supported Cilium telemetry, flow diagnostics, service state and operational visibility.

Core engineering principle: AKS networking must be understood as an integrated system. Kubernetes decides the intended workload and Service state; Azure provides infrastructure connectivity; and the configured node datapath determines how packets are processed and forwarded.

Network Insight · Engineering Guide 22 · Interactive Contents

Explore the Architecture.
Engineer the Network.

A complete engineering journey through Azure Kubernetes Service networking — from pod connectivity and Azure CNI Overlay to Cilium eBPF, Azure VNets, ingress, Zero Trust security, private clusters, high availability and production troubleshooting.

9 Engineering Sections Azure CNI Overlay Cilium eBPF Zero Trust Production AKS
AZURE VNET Routing · Subnets Private Connectivity INGRESS Azure Load Balancer Services · Gateway AKS KUBERNETES CLUSTER NETWORKING FABRIC CILIUM eBPF Policy · Forwarding Pod Datapath ZERO TRUST Network Policies Private Endpoints
ENGINEERING GUIDE READING PROGRESS 0%
Navigate the Engineering Architecture

Nine Sections. One Complete AKS Networking System.

Select any section to jump directly into the architecture. The active section is highlighted automatically as you read.

SECTION 01
AKS Networking Architecture Fundamentals

Managed control plane, worker nodes, Kubernetes networking layers and the complete AKS connectivity model.

SECTION 02
Azure CNI Overlay and Pod IP Addressing

Pod CIDRs, node networking, overlay encapsulation, address planning and pod-to-pod packet delivery.

SECTION 03
AKS Routing, Azure VNets and Hybrid Connectivity

Hub-and-spoke routing, VNet peering, UDRs, Azure Firewall and hybrid enterprise connectivity.

SECTION 04
Services, Azure Load Balancers and Ingress

ClusterIP, NodePort, LoadBalancer, ingress controllers, Gateway API and backend selection.

SECTION 05
AKS Network Security and Zero Trust

Cilium network policies, namespace isolation, default-deny security, NSGs and controlled egress.

SECTION 06
Private AKS Clusters, DNS and Private Endpoints

Private API server access, Azure Private DNS, Private Link, DNS forwarding and hybrid management.

SECTION 07
AKS Cilium eBPF Datapath and Observability

Kernel hooks, Service forwarding, endpoint identities, policy enforcement and packet diagnostics.

SECTION 08
AKS High Availability and Multi-Region Networking

Availability zones, node pools, Azure Front Door, regional failover and disaster recovery.

SECTION 09
AKS Production Troubleshooting and Performance

DNS failures, SNAT exhaustion, MTU mismatches, policy drops, Azure routing and incident investigation.

NETWORK INSIGHT · THE VISUAL AGE
Engineering Guide 22 · Azure Kubernetes Service Networking Architecture
Engineering Guide 22 · Editorial Section 01

AKS Networking Architecture Fundamentals

Azure Kubernetes Service combines the Kubernetes networking model with Azure virtual networking infrastructure to deliver scalable, secure and highly available container networking.

Understanding AKS networking requires engineers to separate the managed Kubernetes control plane, the worker-node infrastructure, and the pod and Service datapath.

These components interact continuously, but they perform different functions. The control plane maintains desired cluster state; Azure infrastructure supplies network connectivity; and the selected container networking implementation determines how application packets are processed.

01.1 / AKS Network Architecture

The Relationship Between Azure Infrastructure and Kubernetes

AKS is a managed Kubernetes platform. Microsoft operates the managed control-plane components, while worker nodes run in Azure infrastructure associated with the cluster.

The networking architecture must support several distinct communication paths: control-plane management, pod-to-pod traffic, Service forwarding, application ingress, outbound internet access and connectivity to private enterprise networks.

AKS ARCHITECTURE — AZURE INFRASTRUCTURE AND KUBERNETES NETWORKING AKS Managed Control Plane API Server · Scheduler · Cluster State AZURE VIRTUAL NETWORK — AKS NODE SUBNET AKS Worker Node A Pod A Application Workload Pod B Application Workload Azure CNI / Cilium Datapath AKS Worker Node B Pod C Application Workload Pod D Application Workload Azure CNI / Cilium Datapath Azure Load Balancer Ingress / Kubernetes Services Azure Routing VNet · UDR · Connectivity Azure Firewall / NAT Controlled Outbound Traffic
Figure 1 — Conceptual AKS network architecture. The control-plane connection represents management communication, not the application data path. Actual packet forwarding depends on the Azure CNI mode, Service configuration and routing design.
01.2 / Managed Control Plane

Why the AKS Control Plane Is Separate From the Workload Datapath

The Kubernetes control plane maintains the desired state of the cluster. It manages workload scheduling, API operations, resource definitions and controller-driven reconciliation.

In AKS, Microsoft manages these control-plane components. Worker nodes communicate with the API server, but normal application traffic between pods does not need to traverse the API server.

COMPONENT 01
Kubernetes API Server

Provides the API through which cluster resources are created, updated and observed. API access may be configured through supported public or private connectivity models.

COMPONENT 02
Scheduler and Controllers

Coordinate desired cluster state, including workload placement and reconciliation. They do not forward application packets.

COMPONENT 03
Worker Nodes

Host application pods and implement workload networking through the configured CNI, Linux kernel and supporting networking components.

Engineering distinction: A private AKS API server changes how the cluster is administered. It does not automatically make Kubernetes application Services private. API-server exposure and application ingress must be designed independently.

01.3 / Azure Virtual Networks

VNets, Node Subnets and Address Planning

AKS worker nodes connect to Azure virtual networking infrastructure. The node subnet provides addresses and connectivity for node interfaces, while pod address allocation depends on the selected networking mode.

Engineers must plan address ranges for nodes, pods, Kubernetes Services, private endpoints and connected enterprise networks. Overlapping CIDRs can complicate or prevent connectivity across VNets and hybrid networks.

Address DomainPurposeEngineering Consideration
VNet address spaceAzure network addressing boundary.Avoid overlap with connected VNets and on-premises networks.
Node subnetAddresses for AKS node interfaces and related infrastructure.Allow for scaling, upgrades and temporary surge nodes.
Pod CIDR / pod subnetAddresses assigned to application pods.Allocation and routability depend on the Azure CNI mode.
Service CIDRVirtual IP range for Kubernetes Services.Keep separate from node, pod and connected network ranges.
Private endpoint subnetPrivate connectivity to supported Azure services.Plan DNS resolution, routing and access controls.
01.4 / Azure CNI Networking

Azure CNI Overlay Versus Pod Subnet Networking

Azure CNI supports different pod addressing architectures. The distinction between overlay and VNet-integrated pod addressing affects IP consumption, routing and how external systems communicate with pods.

CapabilityAzure CNI OverlayAzure CNI Pod Subnet
Pod addressesAllocated from an overlay CIDR separate from the VNet.Allocated from a VNet-integrated pod subnet.
VNet IP consumptionPods do not each consume a VNet address.Pods consume addresses from the configured pod subnet.
Cross-node forwardingUses the supported overlay transport between nodes.Uses the configured VNet-integrated forwarding model.
External reachabilityOverlay pod IPs are not directly routable from external networks by default.Pod IPs participate in the VNet addressing model, subject to routing and security.
Planning emphasisOverlay CIDR sizing and node capacity.Pod subnet capacity and address allocation strategy.

Neither architecture eliminates the need to understand Service routing, NAT, security policies and external connectivity. The correct choice depends on application requirements, network scale and integration constraints.

01.5 / Cilium Integration

Where Cilium eBPF Fits Into AKS Networking

Azure CNI Powered by Cilium integrates supported Cilium eBPF datapath functionality into AKS. This can include network policy enforcement and service handling, depending on the deployment configuration and supported platform capabilities.

Cilium does not replace Azure VNets, Azure routing infrastructure or the managed AKS control plane. Instead, it participates in the node-level networking implementation.

DATAPATH
Packet Processing

Supported eBPF programs process traffic at Linux networking hooks and use kernel maps to support forwarding decisions.

SECURITY
Network Policies

Supported policy enforcement controls communication between workloads based on Kubernetes and Cilium networking constructs.

OPERATIONS
Observability

Operational visibility depends on the telemetry and diagnostic features exposed by the installed AKS networking implementation.

Interactive Architecture-Layer Explorer

Which Component Owns the Network Decision?

Select a layer to inspect its responsibility in the AKS architecture.

Managed Kubernetes Control Plane

The AKS control plane maintains Kubernetes API state and coordinates workload scheduling and reconciliation. It is not the normal forwarding path for pod application traffic.

API Operations → Cluster State → Node Configuration / Workload State
01.6 / Production Engineering

Design Decisions That Shape AKS Network Reliability

Production AKS networking should be designed around application communication requirements rather than treating the default cluster deployment as a complete enterprise network architecture.

Design DecisionWhy It Matters
Azure CNI networking modeDetermines pod addressing, routing characteristics and IP consumption.
Node subnet sizingControls capacity for node growth, upgrades and autoscaling.
Ingress architectureDefines how internal and external clients reach applications.
Outbound connectivityDetermines NAT behaviour, egress identity and firewall inspection paths.
Private API connectivityControls how authorised systems administer the cluster.
Network security policiesControls permitted workload communication and network boundaries.
ObservabilityProvides evidence for packet drops, policy failures and routing incidents.
01.7 / Initial Operational Validation

Inspecting a Deployed AKS Cluster

Before troubleshooting individual packet flows, confirm the cluster networking configuration and the current Kubernetes resource state.

# Inspect AKS network configuration az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query networkProfile # Inspect Kubernetes nodes kubectl get nodes -o wide # Inspect pod placement and addresses kubectl get pods -A -o wide # Inspect Services kubectl get services -A # Inspect Service backends kubectl get endpointslices -A # Inspect Kubernetes network policies kubectl get networkpolicies -A # Inspect system components kubectl get pods -n kube-system # Inspect network-related node details kubectl describe node <node-name>

Compare the observed configuration with the intended design. Confirm pod address ranges, node subnet connectivity, Service CIDRs, cluster DNS and the configured outbound connectivity method.

Production engineering principle: Always identify the selected AKS network plugin, network plugin mode and datapath before applying troubleshooting guidance. Commands, supported features and forwarding behaviour can differ between Azure CNI configurations and AKS releases.

01.8 / Section Summary

The AKS Networking Mental Model

AKS networking becomes easier to reason about when the architecture is separated into clear responsibilities:

Azure provides infrastructure connectivity. Kubernetes defines workloads and Services. Azure CNI integrates pod networking with the selected addressing model. Cilium, when enabled, implements supported eBPF datapath capabilities. Network policies and Azure security controls enforce their respective boundaries.

This foundation is essential before investigating packet forwarding, load balancing, private connectivity or production failures.

Network Insight · Engineering Guide 22 · Lab 01

Azure Kubernetes Service Architecture Explorer

Explore how the Azure-managed Kubernetes control plane, AKS worker nodes, pod networks, Services, Azure VNets, and cloud infrastructure connect to form a production application platform.

Interactive AKS Technical Architecture

Select any component to inspect its engineering role. Run a traffic test to illuminate the relevant path.

AZURE SUBSCRIPTION · AKS NETWORKING ARCHITECTURE AZURE-MANAGED KUBERNETES CONTROL PLANE AKS API SERVER Managed control plane SCHEDULING / STATE Azure-managed services IDENTITY / ACCESS RBAC / Entra integration CUSTOMER AZURE VNET · 10.10.0.0/16 AKS NODE SUBNET · 10.10.1.0/24 WORKER NODE A 10.10.1.4 Frontend Pod 10.244.0.10 CNI / Datapath WORKER NODE B 10.10.1.5 Backend Pod 10.244.1.20 CNI / Datapath AZURE LOAD BALANCER Service type LoadBalancer AZURE ROUTING / EGRESS UDR / Firewall / NAT EXTERNAL / HYBRID Internet · VPN · ExpressRoute
Azure infrastructure Kubernetes nodes Forwarding path Failure
OverlayCNI mode
2Worker nodes
100%Subsystem health
READYTest result
Select a component to inspect its architecture role.

AKS Networking Configuration

Infrastructure and Service Health
Node subnet connectivity
Service backends available
Network policy permits traffic
API endpoint and DNS reachable
Outbound routing available
Select a networking configuration and trace traffic.

AKS Networking State Inspector

LayerResourceState

Packet Path Investigation

Select a traffic path and run the simulation.

AKS Engineering Diagnostic Console

[NETWORK INSIGHT · AKS LAB 01] Cluster: aks-production-demo Region: illustrative Azure region Worker nodes: 2 Pod networking: Azure CNI Overlay Awaiting packet-path investigation.

Architecture and Root Cause Analysis

Azure infrastructure: Provides VNet connectivity, subnets, routing and load balancing.
Kubernetes networking: Provides pod addressing, Services and endpoint selection.
Cilium datapath: Where selected, provides supported eBPF-based forwarding and policy functionality.

Production Engineering Notes

In Azure CNI Overlay, pods use addresses from an overlay CIDR separate from the Azure VNet node subnet. Traffic leaving the cluster toward VNet or external destinations generally uses node-level address translation as appropriate to the traffic path.

In Azure CNI Pod Subnet mode, pod addresses are allocated from an Azure VNet pod subnet. This changes address planning, routing, and integration requirements compared with overlay networking.

A private AKS cluster uses a private API server endpoint and requires suitable private network connectivity and DNS resolution for management access. This is separate from whether application Services are public or private.

Azure CNI Powered by Cilium provides an eBPF-based datapath for supported AKS configurations. Specific features and combinations depend on current AKS networking support and cluster configuration.

Educational simulation. Topology, IP addresses, failure states, and packet traces are illustrative. This widget does not provision Azure resources or connect to a live AKS cluster.

Engineering Guide 22 · Editorial Section 02

Azure CNI Overlay and Pod IP Addressing

Azure CNI networking determines how Kubernetes pods receive IP addresses, how packets cross node boundaries and how workloads communicate with Azure VNets and external networks.

In Azure CNI Overlay, pods receive addresses from a separate overlay address space rather than consuming individual addresses from the Azure VNet subnet. Nodes remain connected through their Azure VNet interfaces.

Understanding this separation is fundamental to designing scalable AKS clusters and troubleshooting cross-node connectivity, routing and source NAT behaviour.

02.1 / Overlay Networking Architecture

How Azure CNI Overlay Separates Pod and Node Addressing

Azure CNI Overlay uses a pod address range that is logically separate from the Azure VNet address space. Each node receives a pod CIDR allocation, while its node interface uses an IP address from the configured Azure subnet.

Cross-node pod traffic is transported between nodes through the overlay implementation. The underlay network needs connectivity between node addresses, while the container networking layer handles the pod-to-pod path.

AZURE CNI OVERLAY — POD CIDRS AND NODE UNDERLAY TRANSPORT AZURE VNET · NODE SUBNET 10.10.1.0/24 AKS Worker Node A Node IP: 10.10.1.10 POD CIDR 192.168.0.0/24 Pod A 192.168.0.10 Pod B 192.168.0.11 AKS Worker Node B Node IP: 10.10.1.11 POD CIDR 192.168.1.0/24 Pod C 192.168.1.10 Pod D 192.168.1.11 Node-to-Node Overlay Transport Node Source NAT Pod Source → Node IP VNet / External Destination Routing and Security Apply
Figure 2 — Illustrative addressing only. The /24 per-node pod CIDR represents a typical Azure CNI Overlay allocation; verify the supported allocation model and networking configuration for the deployed AKS release. Source NAT applies to relevant destinations outside the overlay, subject to platform behaviour and configuration.
02.2 / Address Allocation

Pod CIDRs, Node IPs and Service Address Space

AKS networking uses multiple IP address domains. Engineers should avoid treating the node subnet, pod range and Kubernetes Service range as interchangeable.

Address DomainIllustrative CIDRPurpose
Azure VNet10.10.0.0/16Azure network infrastructure.
AKS node subnet10.10.1.0/24Node interfaces and associated infrastructure.
Azure CNI Overlay pod range192.168.0.0/16Pod addresses allocated from the overlay range.
Node A pod CIDR192.168.0.0/24Example pod allocation assigned to Node A.
Node B pod CIDR192.168.1.0/24Example pod allocation assigned to Node B.
Kubernetes Service CIDR172.20.0.0/16Virtual addresses used by Kubernetes Services.

These ranges are illustrative rather than recommended defaults. Actual CIDRs must be planned against enterprise addressing, supported AKS requirements, connected networks and expected scale.

02.3 / Cross-Node Packet Forwarding

Tracing Pod Traffic Between AKS Nodes

Consider a packet sent from Pod A on Node A to Pod C on Node B. The packet enters the configured node datapath, which determines the destination pod location and forwards traffic through the overlay transport.

ILLUSTRATIVE CROSS-NODE FLOW SOURCE: Pod A IP: 192.168.0.10 Node A: 10.10.1.10 DESTINATION: Pod C IP: 192.168.1.10 Node B: 10.10.1.11 LOGICAL PACKET PATH: 1. Pod A generates packet. 2. Node A datapath identifies destination pod CIDR. 3. Overlay transport carries traffic between node IPs. 4. Node B receives traffic. 5. Node B datapath delivers packet to Pod C. IMPORTANT: Pod addresses and node addresses belong to separate address domains.

The exact implementation depends on the selected Azure CNI mode and datapath. Engineers should inspect actual interfaces, routes and packet captures rather than assuming every AKS configuration uses the same Linux forwarding path.

02.4 / Source NAT

Why External Destinations Often See the Node IP

With Azure CNI Overlay, traffic leaving the overlay for destinations outside the pod CIDR is generally source-NATed to the node IP, subject to supported exceptions and network configuration.

This distinction matters for firewall rules, connection logging, application allowlists and troubleshooting. An external destination may see a node address or a further translated outbound address rather than the original pod address.

STAGE 01
Pod Source Address

The application originates traffic using its assigned overlay IP address.

STAGE 02
Node Translation

The supported outbound datapath translates the source address for traffic leaving the overlay.

STAGE 03
Azure Egress

Additional Azure NAT, firewall or routing infrastructure may determine the final source address visible externally.

Operational distinction: Pod-to-pod overlay forwarding and outbound internet egress are different traffic cases. Do not assume that a packet follows the same NAT or routing path simply because it originates from the same pod.

02.5 / Networking Mode Comparison

Azure CNI Overlay Versus VNet-Integrated Pod Subnets

Engineering PropertyAzure CNI OverlayAzure CNI Pod Subnet
Pod addressingSeparate overlay CIDR.VNet-integrated pod subnet.
VNet IP consumptionPod IPs do not individually consume VNet addresses.Pod IPs consume addresses from the configured subnet.
Cross-node communicationUses the supported overlay transport.Uses the configured VNet-integrated networking model.
External access to pod IPsNot directly routable from external networks by default.Potentially routable within connected networks, subject to routing and security.
Capacity planningOverlay CIDR and node allocations.Pod subnet allocation and address availability.
Primary design tradeoffAddress efficiency versus overlay constraints.VNet integration versus VNet IP consumption.
Interactive Network Mode Explorer

Understand AKS Addressing and Forwarding Decisions

Select a scenario to inspect the expected logical packet path.

Azure CNI Overlay Cross-Node Traffic

Pods use addresses from the overlay CIDR. The configured node datapath transports cross-node traffic using node-level connectivity while preserving the logical pod-to-pod communication model.

Source Pod → Node A Datapath → Overlay Transport → Node B Datapath → Destination Pod
02.6 / Capacity Planning

Designing Pod Address Space for Cluster Growth

Pod addressing must account for current workloads, cluster autoscaling, node pool expansion and upgrade operations.

For Azure CNI Overlay configurations that allocate a /24 pod CIDR per node, a /16 overlay range provides 256 distinct /24 blocks in arithmetic terms. This is not a guarantee of 256 usable production nodes: platform limits, reservations and operational capacity must also be considered.

ILLUSTRATIVE OVERLAY ADDRESS PLANNING Pod CIDR: 192.168.0.0/16 Per-node allocation: /24 Theoretical /24 allocations: 2^(24 - 16) = 256 EXAMPLE DESIGN TARGET: 80 steady-state nodes + autoscaling headroom + upgrade surge capacity + future node pools CHECK: AKS platform limits maxPods configuration Node pool scaling limits Address range overlap Network plugin compatibility

Capacity engineering: Address space should be sized for the cluster's lifecycle, not just its initial node count. Validate current Azure CNI Overlay constraints and AKS version-specific scaling limits before deployment.

02.7 / Production Troubleshooting

Investigating Pod Addressing and Overlay Connectivity

SymptomPossible CauseInvestigation
Pod remains PendingScheduling or capacity problem.Inspect pod events, node resources and scheduling constraints.
Pod sandbox creation failsCNI or IP allocation failure.Inspect pod events and networking component logs.
Same-node works, cross-node failsOverlay transport or node reachability problem.Validate node IP connectivity and network restrictions.
Pod cannot reach external destinationSNAT, Azure routing, NSG or egress failure.Trace node and Azure outbound paths.
Only certain destination ranges failOverlapping CIDRs or route conflicts.Compare pod, Service, VNet and enterprise ranges.
AKS NETWORKING INVESTIGATION # Inspect AKS network profile az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query networkProfile # Pod placement and IP addresses kubectl get pods -A -o wide # Node addresses kubectl get nodes -o wide # Pod events and sandbox failures kubectl describe pod \ <pod-name> \ -n <namespace> # Inspect networking components kubectl get pods -n kube-system # Inspect node network details kubectl describe node <node-name> # On an authorised node shell: ip addr ip route ip -s link # Additional packet capture or # Cilium diagnostics depend on # AKS networking mode and access.

Do not change routes or restart networking components until the failure domain is identified. First establish whether the problem affects one pod, one node, a node pool or the entire cluster.

02.8 / Engineering Summary

The Azure CNI Overlay Mental Model

Pods have overlay addresses. Nodes have Azure VNet addresses. The configured datapath connects those two addressing domains.

Cross-node forwarding depends on node connectivity and the overlay transport, while communication outside the overlay introduces additional routing and NAT considerations.

Understanding these boundaries makes it easier to design scalable clusters and investigate failures involving pod addressing, Azure routing and external connectivity.

Network Insight · Engineering Guide 22 · Lab 02

Azure CNI Packet Path Simulator

Investigate Azure CNI Overlay and Pod Subnet networking. Trace packets across AKS worker nodes, inspect encapsulation and address translation, and evaluate IP allocation, routing and MTU behavior.

Azure CNI Packet Architecture

The diagram illustrates node transport and pod addressing. Overlay encapsulation and VNet-routable pod addresses represent different forwarding architectures.

AZURE VNET · 10.10.0.0/16 AKS NODE SUBNET · 10.10.1.0/24 WORKER NODE A WORKER NODE B 10.10.1.4 10.10.1.5 SOURCE POD DESTINATION POD 10.244.0.10 10.244.1.20 Overlay Transport AZURE NODE NETWORK VNet transport VNET / EXTERNAL EGRESS SNAT / Routing / NAT Gateway
Overlay Networking mode
2 × /24 Pod address blocks
1450 Effective path MTU
READY Packet verdict
Select a networking mode and run the packet simulation to inspect source, destination and transport headers.

Packet Engineering Console

Datapath Health
Node transport routing healthy
Required outbound SNAT available
Pod address allocation available
Configure the packet path and execute a test.

Pod IP Allocation Calculator

ParameterCalculated Value

Packet Processing Pipeline

1. Pod IP allocation
2. Source pod packet creation
3. CNI forwarding decision
4. Node transport / SNAT
5. Destination delivery

Azure CNI Diagnostic Terminal

[NETWORK INSIGHT · AKS LAB 02] Network mode: Azure CNI Overlay Node A: 10.10.1.4 Node B: 10.10.1.5 Awaiting packet simulation.

Packet Header Transformation Inspector

Original packet: Pod source IP → destination pod or external IP.
Overlay mode: Cross-node traffic uses an encapsulated node-to-node transport path.
Pod Subnet mode: Pod IPs are allocated from an Azure VNet subnet.

Production Engineering Notes

Azure CNI Overlay assigns pod addresses from an overlay address space separate from the node subnet. Cross-node pod communication uses an encapsulated transport between nodes. For traffic leaving the cluster, source address translation may use the node IP before Azure outbound networking applies further translation.

Azure CNI Pod Subnet allocates pod addresses from a VNet subnet. Pod address capacity and subnet planning differ from overlay networking. Pod Subnet implementations may use different allocation approaches, including dynamic IP allocation or static block allocation, depending on configuration.

Effective MTU must account for encapsulation overhead, underlying network MTU and the selected datapath. Packet loss caused by MTU mismatch should be investigated using appropriately sized probes and path-level diagnostics.

Educational simulation only. Address ranges, allocation figures, SNAT behavior and packet transformations are simplified to illustrate architectural differences. The calculator does not replace AKS capacity-planning documentation.

Engineering Guide 22 · Editorial Section 03

AKS Routing, Azure VNets and Hybrid Connectivity

AKS networking extends beyond pod communication. Production clusters must exchange traffic with Azure services, connected VNets, enterprise data centres, private endpoints and external networks.

Azure provides the infrastructure routing domain through virtual networks, system routes, user-defined routes, VNet peering and hybrid gateways. Kubernetes and the selected CNI implementation handle workload networking within that infrastructure.

The engineering challenge is to understand how these routing layers interact without introducing asymmetric paths, unreachable address ranges, unexpected source NAT or security-policy conflicts.

03.1 / Enterprise Routing Architecture

AKS in a Hub-and-Spoke Azure Network

A common enterprise design places AKS in a spoke VNet connected to a shared networking hub. The hub can provide centralised firewall inspection, hybrid gateways, DNS infrastructure and connectivity to other application networks.

This architecture separates application infrastructure from shared network services. However, VNet peering alone does not create automatic transitive routing between every connected spoke.

AKS HUB-AND-SPOKE — HYBRID CLOUD NETWORK ARCHITECTURE APPLICATION SPOKE VNET 10.20.0.0/16 AKS Cluster Node Subnet · Azure CNI Pods · Services · Cilium AKS Subnet Route Table System Routes / UDRs SHARED CONNECTIVITY HUB VNET Azure Firewall Central Inspection Hybrid Gateway ExpressRoute / VPN Shared Network Services DNS · Routing · Security · Connectivity Private Azure Services Private Endpoints / PaaS Enterprise Data Centre On-Premises Networks Other Azure Spokes Application / Data VNets Peering + Routes
Figure 3 — Conceptual hub-and-spoke AKS network. Animated paths represent possible logical connectivity, not a single simultaneously active forwarding path. Route tables, gateway transit, firewall next hops and return routes must be explicitly validated.
03.2 / Azure Route Selection

System Routes, User-Defined Routes and Effective Routing

Azure virtual networking uses system routes and can incorporate user-defined routes and routes propagated through supported gateway configurations.

When routing decisions are made, longest-prefix matching is fundamental. For routes with identical address prefixes, Azure applies its documented route-source priority rules.

ROUTE SOURCE 01
System Routes

Azure supplies default routes for VNet connectivity and other platform-defined destinations.

ROUTE SOURCE 02
User-Defined Routes

Custom routes can direct selected traffic toward supported next hops such as Azure Firewall or a network virtual appliance.

ROUTE SOURCE 03
Gateway-Propagated Routes

Hybrid connectivity can introduce routes learned through supported gateway and BGP configurations.

Route TypeExampleEngineering Purpose
System VNet route10.20.0.0/16 → Virtual networkLocal VNet connectivity.
UDR default route0.0.0.0/0 → Firewall private IPControlled outbound traffic.
Specific UDR10.50.0.0/16 → Virtual applianceSelected enterprise network forwarding.
Gateway route172.16.0.0/16 → Virtual network gatewayHybrid network reachability.
Peering route10.30.0.0/16 → VNet peeringConnected VNet address-space reachability.

Important: A UDR on the AKS node subnet influences Azure infrastructure forwarding for applicable traffic. It does not replace Kubernetes Service forwarding, CNI pod routing or network policy enforcement.

03.3 / VNet Peering

Connecting AKS Spokes to Shared Azure Infrastructure

VNet peering allows connected Azure virtual networks to communicate over Azure's private network infrastructure, subject to routing and security configuration.

A common deployment connects the AKS spoke to a central hub containing Azure Firewall, DNS resolvers, hybrid gateways and shared services.

Peering ConsiderationProduction Implication
Address-space overlapOverlapping VNet ranges prevent normal peering between those VNets.
Non-transitive peeringSpoke A does not automatically reach Spoke B through a common hub.
Gateway transitRequires supported peering and gateway configuration.
Forwarded trafficMust be permitted and routed appropriately where appliances forward traffic.
Return pathBidirectional routing and security rules must be validated.
03.4 / Azure Firewall and Forced Tunnelling

Routing AKS Egress Through Centralised Security

Enterprises frequently require outbound AKS traffic to pass through centralised inspection infrastructure. This may involve Azure Firewall, a supported network virtual appliance or another approved egress design.

A user-defined route can direct applicable node-subnet traffic to a firewall next hop. The complete design must also account for the selected AKS outbound type, source NAT, platform dependencies and required destination access.

ILLUSTRATIVE EGRESS ROUTING AKS Pod | v Node Datapath / Applicable SNAT | v AKS Node Subnet | v UDR: 0.0.0.0/0 Next hop: Azure Firewall private IP | v Azure Firewall Policy | v Approved External Destination VALIDATE: - AKS outbound type - Required platform endpoints - Firewall rules - DNS resolution - Source NAT behaviour - Return traffic path

Forced-tunnelling and firewall architectures must follow the supported AKS networking requirements. A default route alone does not guarantee that every packet or platform dependency will follow the intended path.

03.5 / Hybrid Connectivity

ExpressRoute, VPN and Enterprise Network Integration

AKS applications often need access to enterprise databases, authentication services, monitoring systems or other workloads hosted outside Azure.

Azure ExpressRoute and site-to-site VPN provide different connectivity options for linking enterprise networks with Azure VNets.

HYBRID OPTION 01
ExpressRoute

Provides private connectivity through an ExpressRoute service provider or supported connectivity arrangement, with routing integrated into the Azure hybrid architecture.

HYBRID OPTION 02
Site-to-Site VPN

Uses encrypted IPsec tunnels over IP connectivity to connect enterprise networks to supported Azure VPN gateway configurations.

HYBRID OPTION 03
Centralised Hub Routing

Provides shared routing and security functions for connected spokes, subject to explicit transit and forwarding configuration.

Overlay consideration: In Azure CNI Overlay, external networks do not normally route directly to pod overlay addresses. Application access should use supported Service, ingress, proxy or other connectivity patterns rather than assuming pod CIDRs are reachable through ExpressRoute or VPN.

03.6 / Interactive Routing Explorer

Trace AKS Traffic Across Azure and Hybrid Networks

Select a traffic scenario to inspect the logical forwarding path and the engineering checks required.

AKS to Local VNet Destination

Traffic from an AKS workload reaches a destination in the connected Azure VNet through the configured CNI datapath and Azure infrastructure routing. Overlay deployments generally apply source NAT for destinations outside the overlay range, subject to supported exceptions.

AKS Pod → Node Datapath → Azure VNet → Destination
03.7 / Routing Failure Analysis

Common AKS Routing Problems in Production

Failure SymptomLikely Investigation AreaEngineering Action
AKS cannot reach another spokeNon-transitive peering or missing transit route.Validate route tables, forwarding and hub configuration.
Hybrid application unreachableMissing gateway route, firewall rule or return path.Check effective routes and enterprise routing.
Only some destinations failPrefix overlap, route specificity or security policy.Compare effective routes and address ranges.
Outbound traffic fails after UDR changeIncorrect next hop or missing platform dependencies.Inspect AKS outbound configuration and firewall logs.
Private endpoint resolves incorrectlyDNS zone or resolver configuration.Validate private DNS resolution from the workload.
Connections fail intermittentlyAsymmetric routing, NAT or stateful inspection.Trace both directions of the flow.
03.8 / Operational Validation

Inspecting AKS and Azure Routing State

# Inspect AKS networking configuration az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query networkProfile # List Azure route tables az network route-table list \ --resource-group <resource-group> \ --output table # Inspect a route table az network route-table show \ --resource-group <resource-group> \ --name <route-table-name> # Inspect VNet peerings az network vnet peering list \ --resource-group <resource-group> \ --vnet-name <vnet-name> \ --output table # Inspect a VM network interface's # effective routes, when supported az network nic show-effective-route-table \ --resource-group <node-resource-group> \ --name <node-nic-name> # Inspect Kubernetes workloads kubectl get pods -A -o wide # Test DNS from an authorised pod kubectl exec -n <namespace> \ <pod-name> -- nslookup <hostname> # Test connectivity using a container # that includes suitable diagnostic tools kubectl exec -n <namespace> \ <pod-name> -- curl -v \ https://<destination-host>

The effective-route command applies to supported Azure VM network interfaces. AKS-managed node interfaces may have access restrictions or lifecycle considerations, so confirm the node resource group and the appropriate supported diagnostic method before running investigations.

Production troubleshooting principle: Always validate the forward and return paths. A route that successfully sends packets toward a destination does not prove that the destination can return traffic through the expected NAT, firewall and hybrid connectivity architecture.

03.9 / Engineering Summary

The AKS Hybrid Routing Mental Model

Kubernetes determines workload connectivity intent. The CNI datapath forwards pod traffic. Azure VNets, route tables, peering, gateways and security appliances determine how applicable traffic moves through the infrastructure.

Reliable enterprise AKS connectivity depends on consistent address planning, supported transit design, correct route selection, DNS resolution, security enforcement and symmetric return-path handling.

Network Insight · Engineering Guide 22 · Lab 03

AKS Routing Path Explorer

Trace AKS application traffic through Azure VNets, user-defined routes, hub-spoke connectivity, Azure Firewall and enterprise hybrid networks. Investigate routing failures, asymmetric paths, security enforcement and next-hop selection.

Azure AKS Hub-Spoke Routing Architecture

Select a network component to inspect its role. Trace traffic between AKS, the Azure hub, and on-premises infrastructure.

AZURE AKS · HUB-SPOKE HYBRID ROUTING AKS SPOKE VNET CONNECTIVITY HUB VNET AKS CLUSTER Pod / Node Networking 10.10.0.0/16 USER-DEFINED ROUTE Next Hop → Firewall AZURE FIREWALL Network Rules / Egress 10.20.0.4 EXPRESSROUTE GATEWAY Private Hybrid Connectivity ON-PREMISES 172.16.0.0/16 VNet Peering Private Circuit
Firewall Selected next hop
3 Illustrative routes
100% Subsystem health
READY Traffic verdict
Select a component to inspect its routing responsibilities.

Azure Routing Engineering Console

Routing and Security State
AKS route table correctly associated
VNet peering connectivity available
Azure Firewall permits traffic
Hybrid circuit or VPN available
Return route correctly configured
Select a routing path and execute a trace.

Azure Effective Route Table

PrefixSourceNext HopState

Routing Decision Pipeline

1. Source node route lookup
2. User-defined route selection
3. VNet peering transport
4. Firewall policy evaluation
5. Hybrid network forwarding
6. Return-path validation

Azure Routing Diagnostic Terminal

[NETWORK INSIGHT · AKS LAB 03] AKS VNet: 10.10.0.0/16 Hub VNet: 10.20.0.0/16 On-premises: 172.16.0.0/16 Awaiting routing investigation.

Root Cause and Path Interpretation

Effective routes: Azure route selection depends on destination prefix specificity and applicable route sources.
Firewall insertion: A UDR can direct supported traffic to an Azure Firewall private IP as a virtual appliance.
Hybrid routing: ExpressRoute or VPN connectivity requires correct route propagation and return paths.

Production Engineering Notes

Azure routing uses longest-prefix matching. For equally specific routes, Azure applies route-source precedence rules. User-defined routes can override system or BGP routes where applicable, but more-specific routes may still win.

VNet peering provides private connectivity between peered VNets, subject to configuration, security rules and route behavior. Peering is not automatically transitive; hub-spoke designs require deliberate routing and gateway-transit configuration.

ExpressRoute and site-to-site VPN can provide hybrid connectivity, but the appropriate gateway, learned prefixes, route propagation, and return routing must all be considered.

Azure Firewall insertion requires careful consideration of source and destination routes, application requirements, security policy and potential asymmetric forwarding. A healthy forward route does not guarantee a healthy return path.

Educational routing simulation. The effective route table and packet traces are illustrative and do not query Azure. Actual routing depends on deployed subnets, route tables, peering settings, gateways, BGP advertisements and Azure platform behavior.

Engineering Guide 22 · Editorial Section 04

AKS Services, Azure Load Balancers and Ingress Architecture

Kubernetes Services provide stable network identities for dynamic application workloads, while Azure load balancing and ingress infrastructure connect those workloads to clients inside and outside the cluster.

AKS application connectivity spans several forwarding layers: Kubernetes Service virtual IPs, backend EndpointSlices, node datapaths, Azure Standard Load Balancer and optional Layer 7 ingress or Gateway implementations.

Understanding these layers is essential for designing reliable application delivery and troubleshooting connection failures, unhealthy backends, incorrect traffic policies and unexpected source NAT.

04.1 / Application Delivery Architecture

How External Traffic Reaches an AKS Application

A common AKS deployment exposes applications through a Kubernetes LoadBalancer Service or an ingress controller fronted by an Azure load balancer.

The exact forwarding path depends on the Service type, ingress implementation, datapath and traffic policy. An ingress controller or Gateway provides application-layer routing, whereas an Azure Standard Load Balancer primarily performs Layer 4 load balancing.

AKS APPLICATION DELIVERY — EXTERNAL CLIENT TO BACKEND POD External Client HTTPS / TCP Request Azure Standard Load Balancer Frontend IP · Backend Pool · Health Probe AKS CLUSTER — KUBERNETES NETWORKING Ingress Controller / Gateway HTTP Routing · TLS · Host / Path Rules Kubernetes Service Service VIP / Backend Selection Backend Pod A Ready Endpoint Backend Pod B Ready Endpoint Backend Pod C Ready Endpoint
Figure 4 — Conceptual application ingress flow. The diagram represents an ingress controller exposed through an Azure LoadBalancer Service. Azure load balancers generally forward to configured node backends; the ingress controller and Kubernetes datapath then process the application request. The three backend lines illustrate eligible destinations, not packet duplication.
04.2 / Kubernetes Service Types

ClusterIP, NodePort and LoadBalancer

Kubernetes Services provide stable network abstractions for workloads whose pod addresses may change during scaling, rescheduling and deployment operations.

SERVICE TYPE 01
ClusterIP

Exposes a Service through a virtual cluster IP for internal connectivity. It does not independently create an externally accessible Azure load balancer.

SERVICE TYPE 02
NodePort

Exposes a Service through a port on eligible node addresses, subject to network reachability and firewall rules.

SERVICE TYPE 03
LoadBalancer

Requests external load-balancing infrastructure through the supported cloud integration. In AKS this commonly provisions Azure Standard Load Balancer resources.

Service TypePrimary Access ModelAKS Networking Implication
ClusterIPInternal Service virtual IP.Handled by the configured Kubernetes Service datapath.
NodePortNode address and allocated Service port.Requires appropriate node reachability and traffic policy.
LoadBalancerAzure load balancer frontend.Requires supported Azure cloud integration, backend configuration and health probes.
ExternalNameDNS alias to an external hostname.DNS mapping rather than a conventional Service proxy or Azure load balancer.
04.3 / Azure Standard Load Balancer

Frontend IPs, Backend Pools and Health Probes

Azure Standard Load Balancer distributes supported Layer 4 traffic using configured frontend IPs, rules, backend pools and health probes.

In AKS, the Kubernetes cloud integration coordinates supported Azure load balancer resources for LoadBalancer Services. The backend architecture and node traffic handling depend on the cluster and Service configuration.

FRONTEND
Public or Internal IP

Defines the address clients use to reach the application. Internal load balancers provide private frontend addresses.

BACKEND
Eligible Node Backends

Azure forwards traffic to healthy backend targets according to the load-balancing configuration.

HEALTH
Health Probe State

Health probes determine backend eligibility. Probe configuration must align with the Service and traffic policy.

Important distinction: An Azure load balancer health probe does not replace Kubernetes application readiness checks. A node may be eligible at the Azure load-balancing layer while individual application endpoints have separate readiness state.

04.4 / Ingress Controllers and Gateway API

Layer 7 Routing Above Kubernetes Services

Ingress controllers provide HTTP and HTTPS routing capabilities such as hostname matching, path routing and TLS termination.

Gateway API provides a more expressive Kubernetes resource model for supported traffic-management implementations. It separates gateway infrastructure from route definitions and supports different ownership and delegation patterns.

CapabilityIngressGateway API
Routing resourcesIngress resources interpreted by a controller.Gateway, HTTPRoute and other supported route resources.
ImplementationRequires an ingress controller.Requires a compatible Gateway API controller.
HTTP routingHost and path routing.Expressive matching and routing based on implementation support.
Role separationTypically controller-specific.Designed for infrastructure and application role separation.
Azure integrationDepends on the selected ingress implementation.Depends on the selected Gateway controller and supported Azure integration.

Neither an Ingress resource nor a Gateway API resource forwards packets by itself. A compatible controller must implement the configuration and operate the associated traffic-processing infrastructure.

04.5 / Backend Selection

EndpointSlices, Readiness and Service Datapath Forwarding

Kubernetes EndpointSlices represent backend endpoints associated with Services. Controllers update endpoint information as pods become ready, scale or terminate.

The configured Kubernetes Service datapath uses this state to forward traffic to eligible destinations. Depending on AKS networking configuration, the implementation may use supported Cilium eBPF mechanisms or other Service forwarding components.

CONCEPTUAL SERVICE FORWARDING Client | v Service VIP | v Service Datapath | +-- Backend Pod A [Ready] | +-- Backend Pod B [Ready] | +-- Backend Pod C [Not Ready] EXPECTED: New traffic is directed toward eligible backends according to Service state and datapath rules. VALIDATE: Service selectors EndpointSlice contents Pod readiness targetPort configuration Network policies Node and load balancer health
04.6 / Traffic Policies

externalTrafficPolicy, Source IP and Local Backends

Kubernetes Service traffic policies influence how external traffic is forwarded and whether the original client source IP can be preserved.

ConfigurationTypical BehaviourEngineering Tradeoff
externalTrafficPolicy: ClusterTraffic may be forwarded to eligible endpoints on other nodes.Broader backend reachability; source IP handling depends on the path.
externalTrafficPolicy: LocalExternal Service traffic uses node-local eligible endpoints.Can preserve client source IP, but requires careful backend and health-probe design.
internalTrafficPolicy: ClusterInternal traffic may use eligible endpoints across the cluster.Supports cluster-wide backend selection.
internalTrafficPolicy: LocalInternal traffic is restricted to eligible node-local endpoints.Traffic may fail when no local eligible endpoint exists.

Production design principle: Do not enable Local traffic policies solely to preserve source IP without understanding node placement, health probes, backend distribution and the behaviour of the selected AKS datapath.

Interactive AKS Application Traffic Explorer

Trace Kubernetes Service and Ingress Traffic

Select a scenario to understand the forwarding layers and operational dependencies.

Internal ClusterIP Service

A pod connects to a Kubernetes Service virtual IP. The configured Service datapath selects an eligible backend from the current endpoint state and forwards the connection to the destination pod.

Client Pod → ClusterIP → Service Datapath → Backend Pod
04.7 / Operational Validation

Inspecting Services, Endpoints and Ingress Resources

# Inspect Kubernetes Services kubectl get services -A # Inspect Service configuration kubectl describe service \ <service-name> \ -n <namespace> # Inspect EndpointSlices kubectl get endpointslices -A # Inspect endpoint details kubectl describe endpointslice \ <endpointslice-name> \ -n <namespace> # Inspect ingress resources kubectl get ingress -A # Inspect ingress configuration kubectl describe ingress \ <ingress-name> \ -n <namespace> # Inspect Gateway API resources # when the CRDs are installed kubectl get gateways -A kubectl get httproutes -A # Inspect Azure load balancers az network lb list \ --resource-group <node-resource-group> \ --output table # Inspect application readiness kubectl get pods \ -n <namespace> \ -o wide

Gateway API commands require the relevant custom resource definitions. Azure load balancer inspection requires access to the correct resource group and sufficient Azure permissions.

04.8 / Production Troubleshooting

Diagnosing Application Delivery Failures

SymptomLikely Investigation AreaRecommended Check
Service has no endpointsSelector mismatch or pod readiness.Inspect EndpointSlices and pod labels.
LoadBalancer IP remains pendingAzure resource provisioning or permissions.Inspect Service events and cloud resource state.
External connections time outNSG, load balancer, backend or policy problem.Check frontend rules, probes and node reachability.
HTTP 404 through ingressHost or path routing mismatch.Inspect ingress rules and controller logs.
HTTP 502 or 503Backend reachability or health problem.Inspect controller logs, Services and endpoint readiness.
Source IP differs from expectationSNAT, proxy or traffic policy.Trace each forwarding layer and headers where applicable.
Intermittent application failuresBackend churn, health probes or uneven placement.Correlate endpoint changes with load balancer health.

Engineering troubleshooting rule: Separate Layer 4 reachability from Layer 7 application routing. A healthy Azure load balancer does not prove that an ingress route, Kubernetes Service or backend application is functioning correctly.

04.9 / Engineering Summary

The AKS Application Delivery Mental Model

Azure load balancers provide infrastructure-level traffic distribution. Kubernetes Services provide stable workload connectivity. Ingress controllers and Gateway implementations provide application-layer routing. EndpointSlices and the configured datapath connect Services to eligible pods.

Production reliability depends on correct frontend configuration, backend health, readiness, traffic policies, network security and observability across every layer.

Engineering Guide 22 · Editorial Section 04

AKS Services, Azure Load Balancers and Ingress Architecture

Kubernetes Services provide stable network identities for dynamic application workloads, while Azure load balancing and ingress infrastructure connect those workloads to clients inside and outside the cluster.

AKS application connectivity spans several forwarding layers: Kubernetes Service virtual IPs, backend EndpointSlices, node datapaths, Azure Standard Load Balancer and optional Layer 7 ingress or Gateway implementations.

Understanding these layers is essential for designing reliable application delivery and troubleshooting connection failures, unhealthy backends, incorrect traffic policies and unexpected source NAT.

04.1 / Application Delivery Architecture

How External Traffic Reaches an AKS Application

A common AKS deployment exposes applications through a Kubernetes LoadBalancer Service or an ingress controller fronted by an Azure load balancer.

The exact forwarding path depends on the Service type, ingress implementation, datapath and traffic policy. An ingress controller or Gateway provides application-layer routing, whereas an Azure Standard Load Balancer primarily performs Layer 4 load balancing.

AKS APPLICATION DELIVERY — EXTERNAL CLIENT TO BACKEND POD External Client HTTPS / TCP Request Azure Standard Load Balancer Frontend IP · Backend Pool · Health Probe AKS CLUSTER — KUBERNETES NETWORKING Ingress Controller / Gateway HTTP Routing · TLS · Host / Path Rules Kubernetes Service Service VIP / Backend Selection Backend Pod A Ready Endpoint Backend Pod B Ready Endpoint Backend Pod C Ready Endpoint
Figure 4 — Conceptual application ingress flow. The diagram represents an ingress controller exposed through an Azure LoadBalancer Service. Azure load balancers generally forward to configured node backends; the ingress controller and Kubernetes datapath then process the application request. The three backend lines illustrate eligible destinations, not packet duplication.
04.2 / Kubernetes Service Types

ClusterIP, NodePort and LoadBalancer

Kubernetes Services provide stable network abstractions for workloads whose pod addresses may change during scaling, rescheduling and deployment operations.

SERVICE TYPE 01
ClusterIP

Exposes a Service through a virtual cluster IP for internal connectivity. It does not independently create an externally accessible Azure load balancer.

SERVICE TYPE 02
NodePort

Exposes a Service through a port on eligible node addresses, subject to network reachability and firewall rules.

SERVICE TYPE 03
LoadBalancer

Requests external load-balancing infrastructure through the supported cloud integration. In AKS this commonly provisions Azure Standard Load Balancer resources.

Service TypePrimary Access ModelAKS Networking Implication
ClusterIPInternal Service virtual IP.Handled by the configured Kubernetes Service datapath.
NodePortNode address and allocated Service port.Requires appropriate node reachability and traffic policy.
LoadBalancerAzure load balancer frontend.Requires supported Azure cloud integration, backend configuration and health probes.
ExternalNameDNS alias to an external hostname.DNS mapping rather than a conventional Service proxy or Azure load balancer.
04.3 / Azure Standard Load Balancer

Frontend IPs, Backend Pools and Health Probes

Azure Standard Load Balancer distributes supported Layer 4 traffic using configured frontend IPs, rules, backend pools and health probes.

In AKS, the Kubernetes cloud integration coordinates supported Azure load balancer resources for LoadBalancer Services. The backend architecture and node traffic handling depend on the cluster and Service configuration.

FRONTEND
Public or Internal IP

Defines the address clients use to reach the application. Internal load balancers provide private frontend addresses.

BACKEND
Eligible Node Backends

Azure forwards traffic to healthy backend targets according to the load-balancing configuration.

HEALTH
Health Probe State

Health probes determine backend eligibility. Probe configuration must align with the Service and traffic policy.

Important distinction: An Azure load balancer health probe does not replace Kubernetes application readiness checks. A node may be eligible at the Azure load-balancing layer while individual application endpoints have separate readiness state.

04.4 / Ingress Controllers and Gateway API

Layer 7 Routing Above Kubernetes Services

Ingress controllers provide HTTP and HTTPS routing capabilities such as hostname matching, path routing and TLS termination.

Gateway API provides a more expressive Kubernetes resource model for supported traffic-management implementations. It separates gateway infrastructure from route definitions and supports different ownership and delegation patterns.

CapabilityIngressGateway API
Routing resourcesIngress resources interpreted by a controller.Gateway, HTTPRoute and other supported route resources.
ImplementationRequires an ingress controller.Requires a compatible Gateway API controller.
HTTP routingHost and path routing.Expressive matching and routing based on implementation support.
Role separationTypically controller-specific.Designed for infrastructure and application role separation.
Azure integrationDepends on the selected ingress implementation.Depends on the selected Gateway controller and supported Azure integration.

Neither an Ingress resource nor a Gateway API resource forwards packets by itself. A compatible controller must implement the configuration and operate the associated traffic-processing infrastructure.

04.5 / Backend Selection

EndpointSlices, Readiness and Service Datapath Forwarding

Kubernetes EndpointSlices represent backend endpoints associated with Services. Controllers update endpoint information as pods become ready, scale or terminate.

The configured Kubernetes Service datapath uses this state to forward traffic to eligible destinations. Depending on AKS networking configuration, the implementation may use supported Cilium eBPF mechanisms or other Service forwarding components.

CONCEPTUAL SERVICE FORWARDING Client | v Service VIP | v Service Datapath | +-- Backend Pod A [Ready] | +-- Backend Pod B [Ready] | +-- Backend Pod C [Not Ready] EXPECTED: New traffic is directed toward eligible backends according to Service state and datapath rules. VALIDATE: Service selectors EndpointSlice contents Pod readiness targetPort configuration Network policies Node and load balancer health
04.6 / Traffic Policies

externalTrafficPolicy, Source IP and Local Backends

Kubernetes Service traffic policies influence how external traffic is forwarded and whether the original client source IP can be preserved.

ConfigurationTypical BehaviourEngineering Tradeoff
externalTrafficPolicy: ClusterTraffic may be forwarded to eligible endpoints on other nodes.Broader backend reachability; source IP handling depends on the path.
externalTrafficPolicy: LocalExternal Service traffic uses node-local eligible endpoints.Can preserve client source IP, but requires careful backend and health-probe design.
internalTrafficPolicy: ClusterInternal traffic may use eligible endpoints across the cluster.Supports cluster-wide backend selection.
internalTrafficPolicy: LocalInternal traffic is restricted to eligible node-local endpoints.Traffic may fail when no local eligible endpoint exists.

Production design principle: Do not enable Local traffic policies solely to preserve source IP without understanding node placement, health probes, backend distribution and the behaviour of the selected AKS datapath.

Interactive AKS Application Traffic Explorer

Trace Kubernetes Service and Ingress Traffic

Select a scenario to understand the forwarding layers and operational dependencies.

Internal ClusterIP Service

A pod connects to a Kubernetes Service virtual IP. The configured Service datapath selects an eligible backend from the current endpoint state and forwards the connection to the destination pod.

Client Pod → ClusterIP → Service Datapath → Backend Pod
04.7 / Operational Validation

Inspecting Services, Endpoints and Ingress Resources

# Inspect Kubernetes Services kubectl get services -A # Inspect Service configuration kubectl describe service \ <service-name> \ -n <namespace> # Inspect EndpointSlices kubectl get endpointslices -A # Inspect endpoint details kubectl describe endpointslice \ <endpointslice-name> \ -n <namespace> # Inspect ingress resources kubectl get ingress -A # Inspect ingress configuration kubectl describe ingress \ <ingress-name> \ -n <namespace> # Inspect Gateway API resources # when the CRDs are installed kubectl get gateways -A kubectl get httproutes -A # Inspect Azure load balancers az network lb list \ --resource-group <node-resource-group> \ --output table # Inspect application readiness kubectl get pods \ -n <namespace> \ -o wide

Gateway API commands require the relevant custom resource definitions. Azure load balancer inspection requires access to the correct resource group and sufficient Azure permissions.

04.8 / Production Troubleshooting

Diagnosing Application Delivery Failures

SymptomLikely Investigation AreaRecommended Check
Service has no endpointsSelector mismatch or pod readiness.Inspect EndpointSlices and pod labels.
LoadBalancer IP remains pendingAzure resource provisioning or permissions.Inspect Service events and cloud resource state.
External connections time outNSG, load balancer, backend or policy problem.Check frontend rules, probes and node reachability.
HTTP 404 through ingressHost or path routing mismatch.Inspect ingress rules and controller logs.
HTTP 502 or 503Backend reachability or health problem.Inspect controller logs, Services and endpoint readiness.
Source IP differs from expectationSNAT, proxy or traffic policy.Trace each forwarding layer and headers where applicable.
Intermittent application failuresBackend churn, health probes or uneven placement.Correlate endpoint changes with load balancer health.

Engineering troubleshooting rule: Separate Layer 4 reachability from Layer 7 application routing. A healthy Azure load balancer does not prove that an ingress route, Kubernetes Service or backend application is functioning correctly.

04.9 / Engineering Summary

The AKS Application Delivery Mental Model

Azure load balancers provide infrastructure-level traffic distribution. Kubernetes Services provide stable workload connectivity. Ingress controllers and Gateway implementations provide application-layer routing. EndpointSlices and the configured datapath connect Services to eligible pods.

Production reliability depends on correct frontend configuration, backend health, readiness, traffic policies, network security and observability across every layer.

Network Insight · Engineering Guide 22 · Lab 04

AKS Service Traffic Simulator

Investigate Kubernetes Service exposure, Azure Load Balancer forwarding, ingress routing, Gateway API concepts, backend selection, source-IP preservation and application availability.

AKS Application Traffic Architecture

Select any architecture component to inspect its role. The active packet path changes according to the Service exposure mode and simulated failure.

AKS APPLICATION DELIVERY · SERVICE DATAPATH AZURE VNET / AKS CLUSTER CLIENT 198.51.100.20 AZURE LOAD BALANCER Frontend / Health Probe INGRESS / GATEWAY L7 HTTP Routing KUBERNETES SERVICE ClusterIP / NodePort 10.96.10.100:80 WORKER NODE A WORKER NODE B BACKEND POD A 10.244.0.10:8080 BACKEND POD B 10.244.1.20:8080
LoadBalancer Service mode
2 / 2 Ready backends
Cluster External traffic policy
READY Traffic verdict
Select a component to inspect its application delivery responsibilities.

Service Engineering Console

Application Health and Routing
Backend Pod A ready
Backend Pod B ready
Azure LB health probe passes
Ingress / Gateway route valid
Network policy allows application
Configure a Service and simulate a request.

Kubernetes Service and Endpoint State

ResourceAddressRoleState

Application Packet Processing

1. Client initiates application request
2. Azure / Kubernetes exposure path
3. Service backend selection
4. Policy and health validation
5. Backend application response

AKS Service Diagnostic Terminal

[NETWORK INSIGHT · AKS LAB 04] Namespace: production Service: frontend-service ClusterIP: 10.96.10.100 Port: 80 TargetPort: 8080 Awaiting application request.

Traffic Engineering and Root Cause

ClusterIP: Internal virtual Service address used by workloads inside the cluster.
LoadBalancer: Azure infrastructure provides external or internal Service exposure.
Ingress and Gateway API: Application-layer routing requires an appropriate controller implementation.

Production Engineering Notes

A Kubernetes ClusterIP Service provides a stable virtual address for a group of backend endpoints. NodePort exposes a Service through a port on eligible cluster nodes. LoadBalancer Services integrate with Azure Load Balancer when supported by the cluster configuration.

Ingress controllers and Gateway API implementations provide application-level routing capabilities, but the Kubernetes resources alone do not create a working data plane. A controller or gateway implementation must be deployed and configured.

For externalTrafficPolicy: Local, only nodes with suitable local endpoints can serve external traffic. This can preserve the original client source IP for supported paths, but it requires careful attention to Azure Load Balancer health probes, backend distribution and node-level availability.

With externalTrafficPolicy: Cluster, Service traffic may be forwarded to endpoints on other nodes. Client source-IP visibility depends on the actual Service datapath and translation behavior.

Educational browser-only simulation. Backend selection and probe behavior are simplified. Real AKS implementations depend on the selected network datapath, Azure Load Balancer configuration, Kubernetes endpoint readiness and controller behavior.

Engineering Guide 22 · Editorial Section 05

AKS Network Security and Zero Trust Architecture

Securing AKS requires more than protecting the Azure VNet perimeter. Production Kubernetes environments need workload-level segmentation, controlled ingress and egress, identity-aware access and consistent enforcement across Azure and Kubernetes networking layers.

Azure provides infrastructure security through network security groups, Azure Firewall, private connectivity and identity services. Kubernetes network policies and supported Cilium datapath capabilities provide additional workload-level controls.

The central engineering objective is to enforce least-privilege connectivity while preserving the communication paths applications require to function.

05.1 / Zero Trust Security Architecture

Layered Enforcement Across Azure and Kubernetes

AKS network security is a layered architecture. A packet can encounter workload policy enforcement, node networking, Azure subnet controls, routing infrastructure and centralised firewall inspection.

These controls operate at different boundaries. An NSG does not replace a Kubernetes NetworkPolicy, and a Kubernetes NetworkPolicy does not replace enterprise egress inspection.

AKS ZERO TRUST — WORKLOAD SEGMENTATION AND AZURE SECURITY AKS CLUSTER / AZURE VNET Frontend Namespace Frontend Pod app=frontend Backend Namespace Backend API Pod app=backend · TCP 8080 Untrusted Namespace Untrusted Pod No matching allow rule ALLOW DENY Cilium / Kubernetes Network Policy Enforcement Azure NSG Subnet / NIC Security Azure Firewall Central Egress Inspection Approved Destination Private / External Service
Figure 5 — Conceptual layered Zero Trust architecture. Allowed and denied flows are illustrative policy outcomes. Actual enforcement depends on policy selectors, ingress and egress isolation, the supported AKS policy implementation and the applicable Azure network path.
05.2 / Kubernetes Network Policies

Default-Deny and Explicitly Allowed Workload Traffic

Kubernetes NetworkPolicy resources select pods and define permitted ingress and egress communication. Once a pod is isolated for a direction, only traffic allowed by applicable policies is permitted in that direction.

Network policies are additive. Multiple policies selecting the same pod contribute to the set of allowed connections rather than being evaluated as a simple ordered firewall rule list.

# Default-deny ingress and egress apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny namespace: production spec: podSelector: {} policyTypes: - Ingress - Egress

This policy isolates pods in the production namespace for both directions. It should be introduced with explicit allow policies for required DNS, monitoring, application and platform communication.

Policy design warning: Applying default-deny egress without permitting required DNS and application dependencies can cause widespread service failures. Validate the actual traffic requirements before enforcing the policy.

05.3 / Namespace and Workload Segmentation

Allowing Frontend-to-Backend Traffic Without Broad Trust

Namespace labels and pod labels allow engineers to define precise workload communication rules. For example, a backend API can permit requests only from approved frontend workloads.

# Example: allow frontend pods to reach # backend pods on TCP port 8080. # # Assumes the source namespace has: # kubernetes.io/metadata.name: frontend # # Assumes source pods have: # app: frontend # # Assumes destination pods have: # app: backend apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-backend namespace: backend spec: podSelector: matchLabels: app: backend policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: kubernetes.io/metadata.name: frontend podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080

The namespaceSelector and podSelector in the same peer entry are combined, so the source must match both. The policy governs ingress to selected backend pods; a separately isolated frontend also needs an appropriate egress allow rule.

05.4 / Cilium eBPF Security

Network Policy Enforcement in the Node Datapath

Azure CNI Powered by Cilium can enforce supported Kubernetes network policies through its eBPF-based datapath.

Cilium uses endpoint identity and policy information to evaluate permitted traffic. Exact enforcement hooks, observability tools and supported policy features depend on the AKS deployment and platform version.

CONTROL 01
Pod Identity and Labels

Labels and selectors define which workloads a policy applies to and which peers are permitted.

CONTROL 02
eBPF Policy Enforcement

Supported Cilium datapath functions evaluate workload traffic against installed policy state.

CONTROL 03
Policy Observability

Available diagnostics can help distinguish policy drops from routing, DNS or application failures.

AKS compatibility: Do not assume that every upstream Cilium feature or custom Cilium policy resource is supported in managed AKS. Confirm the capabilities and support boundaries of Azure CNI Powered by Cilium for the deployed version.

05.5 / Azure Infrastructure Security

NSGs, Azure Firewall and Private Connectivity

Security LayerPrimary ResponsibilityEngineering Consideration
Kubernetes NetworkPolicyPod-level ingress and egress permissions.Selectors, isolation directions and required application flows.
Cilium datapathSupported workload policy enforcement.AKS compatibility, policy state and available diagnostics.
Azure NSGSubnet and network-interface traffic filtering.Node reachability, ingress, egress and platform requirements.
Azure FirewallCentralised network and application rule enforcement.UDRs, NAT, DNS, rule priorities and return paths.
Private EndpointPrivate IP connectivity to supported Azure services.DNS resolution, service access controls and network reachability.
Private AKS APIPrivate control-plane access.Administrative network access and private DNS.

These mechanisms are complementary. A robust design applies each control to the boundary it can reliably enforce rather than relying on one product to provide complete Zero Trust security.

05.6 / DNS and Egress Security

Why DNS Must Be Included in Network Policy Design

Many Kubernetes workloads depend on cluster DNS for Service discovery and external hostname resolution.

When egress isolation is enabled, DNS traffic must be permitted to the appropriate resolver endpoints. The exact rules depend on the cluster DNS implementation, labels, namespaces and network path.

ILLUSTRATIVE DNS SECURITY PATH Application Pod | v DNS Query | v NetworkPolicy Egress Decision | v CoreDNS / Cluster DNS Service | v Upstream DNS Resolver | v Resolved Destination Address CHECK: - DNS Service address - DNS backend pod labels - UDP/TCP port 53 - Egress policy selectors - Azure DNS forwarding path - Private DNS zone resolution

A policy that permits TCP 443 to an external service may still fail if the application cannot resolve the destination hostname.

Interactive Zero Trust Policy Explorer

Evaluate AKS Workload Communication Scenarios

Select a scenario to inspect the expected policy decision and the relevant enforcement boundary.

Allowed Frontend-to-Backend Traffic

A backend ingress policy permits matching frontend pods from an approved namespace to reach TCP port 8080. The connection also requires source egress permission if the frontend is egress-isolated.

Frontend Pod → Policy Match → ALLOW TCP 8080 → Backend Pod
05.7 / Operational Validation

Inspecting AKS Network Security State

# List Kubernetes network policies kubectl get networkpolicies -A # Inspect policy configuration kubectl describe networkpolicy \ <policy-name> \ -n <namespace> # Inspect namespace labels kubectl get namespaces --show-labels # Inspect workload labels kubectl get pods -A --show-labels # Inspect AKS network profile az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query networkProfile # Inspect Azure NSGs az network nsg list \ --resource-group <resource-group> \ --output table # Inspect Azure Firewall resources az network firewall list \ --resource-group <resource-group> \ --output table # Test application connectivity kubectl exec -n <namespace> \ <pod-name> -- curl -v \ http://<service-name>:8080 # Test DNS where tools are available kubectl exec -n <namespace> \ <pod-name> -- nslookup \ <service-name>

The availability of Cilium-specific diagnostic commands depends on the managed AKS networking configuration. Use supported cluster and Azure diagnostics rather than assuming unrestricted access to upstream Cilium tooling.

05.8 / Security Failure Analysis

Diagnosing Policy and Connectivity Failures

SymptomLikely CauseInvestigation
Backend unreachable after policy changeMissing ingress or source egress allow rule.Inspect policy selectors and both traffic directions.
DNS fails after default-denyDNS egress not permitted.Check resolver endpoints and UDP/TCP 53 permissions.
External API unreachablePolicy, NSG, firewall or routing restriction.Trace workload and Azure security boundaries.
Only one namespace is affectedNamespace selector or label mismatch.Compare actual labels with policy selectors.
Private endpoint connection failsDNS, routing or service access restriction.Resolve the endpoint and validate the private network path.
Unexpected traffic remains allowedAdditional additive policy or missing isolation.Inspect all policies selecting the affected pods.

Security troubleshooting principle: Do not assume that a timeout means a Kubernetes network policy denied the packet. DNS resolution, Service forwarding, node routing, Azure NSGs, firewall rules and application readiness can produce similar symptoms.

05.9 / Engineering Summary

The AKS Zero Trust Mental Model

Workload policies control permitted pod communication. Cilium implements supported datapath enforcement. Azure NSGs and firewalls protect infrastructure boundaries. Private connectivity reduces unnecessary public exposure.

Effective AKS security requires coordinated policy design, correct DNS and routing behaviour, operational visibility and regular validation of least-privilege application flows.

Network Insight · Engineering Guide 22 · Lab 05

AKS Network Policy and Zero Trust Lab

Investigate how Kubernetes workload policy, Cilium eBPF enforcement, Azure NSGs and Azure Firewall combine to protect AKS application traffic.

AKS Layered Security Architecture

Select a security component to inspect its enforcement responsibility. Run a traffic test to identify the first blocking layer.

AKS ZERO TRUST · MULTI-LAYER SECURITY AZURE VNET / AKS CLUSTER FRONTEND POD Namespace: web Identity: frontend API POD Namespace: app TCP 8080 DATABASE POD Namespace: data TCP 5432 CILIUM eBPF Identity / L3-L4 Policy Supported AKS Datapath AZURE NSG Subnet / NIC Rules Azure Network Layer AZURE FIREWALL Central Egress Control Routed Traffic Inspection EXTERNAL DESTINATION Internet / Enterprise API
Zero Trust Policy mode
Cilium Workload enforcement
100% Security health
READY Traffic verdict
Select a workload or security component to inspect its enforcement responsibilities.

Zero Trust Engineering Console

Security Controls
Cross-namespace traffic allowed
Approved destination port allowed
Azure NSG permits modeled flow
Azure Firewall permits egress
Select a workload flow and evaluate its security path.

Security Decision Matrix

LayerEnforcement ScopeDecision

Zero Trust Policy Evaluation Pipeline

1. Identify source and destination workloads
2. Evaluate namespace segmentation
3. Evaluate workload L3/L4 policy
4. Evaluate Azure NSG
5. Evaluate routed Azure Firewall policy
6. Determine allow or deny verdict

AKS Security Diagnostic Terminal

[NETWORK INSIGHT · AKS LAB 05] Security posture: Least Privilege Policy provider: Cilium Namespaces: web / app / data Awaiting security policy evaluation.

Policy Root Cause Analysis

Kubernetes NetworkPolicy: Controls permitted workload ingress and egress where enforced by the CNI provider.
Cilium: Provides supported identity-aware eBPF datapath enforcement.
Azure NSG and Firewall: Apply controls at Azure networking boundaries and routed inspection points.

Production Engineering Notes

Kubernetes NetworkPolicy is an additive allow-policy model. When a pod is isolated for a direction, traffic must match an applicable allow rule. Both egress and ingress enforcement may matter for pod-to-pod communication.

Azure CNI Powered by Cilium uses supported eBPF-based policy enforcement. Kubernetes NetworkPolicy semantics and available Cilium-specific capabilities must be distinguished; advanced Cilium policy features are not necessarily exposed or supported in every managed AKS configuration.

Azure NSGs apply to Azure subnet and network-interface traffic according to Azure networking semantics. They do not replace Kubernetes workload policies. The visibility of individual pod flows at an NSG boundary depends on networking mode and the actual packet path.

Azure Firewall evaluates traffic that is routed through it. A Firewall rule cannot enforce a path that bypasses the firewall. Hub-spoke egress designs must therefore consider route tables, return paths, SNAT behavior and network policy together.

Educational simulation. The security matrix is an illustrative decision model, not a live Cilium policy engine or Azure NSG/Firewall rule evaluation. Actual enforcement depends on deployed policy objects, AKS networking mode and routing.

Engineering Guide 22 · Editorial Section 06

Private AKS Clusters, DNS and Azure Private Endpoints

Private AKS architectures restrict Kubernetes API server connectivity to approved network paths and use Azure private networking to connect applications to internal services and supported Azure PaaS resources.

The critical engineering dependencies are private DNS resolution, API server reachability, VNet routing, hybrid connectivity and correct separation between Kubernetes management traffic and application data traffic.

A private cluster can still contain publicly exposed applications if public ingress is configured. Private control-plane connectivity and private application delivery must therefore be designed independently.

06.1 / Private Cluster Architecture

How Private AKS Connects Management and Application Networks

Private AKS configurations provide private connectivity to the Kubernetes API server using supported Azure networking mechanisms. Cluster nodes and authorised administrators must be able to resolve and reach the configured API endpoint.

Enterprise deployments frequently use hub-and-spoke VNets, central DNS services and ExpressRoute or VPN to extend private management connectivity to corporate environments.

PRIVATE AKS — API SERVER, DNS AND PRIVATE SERVICE CONNECTIVITY AKS SPOKE VNET AKS Worker Nodes Kubelet · Pods · CNI Datapath Private Node Connectivity Private API Connectivity Private API Server Endpoint Managed Control Plane Access SHARED HUB VNET DNS Resolver / Forwarder Private Zone Resolution Hybrid DNS Integration Hybrid Connectivity ExpressRoute / Site-to-Site VPN Enterprise Management Networks Azure Private Endpoint Storage / Database / Supported PaaS Enterprise Administrator kubectl / Management Client
Figure 6 — Conceptual private AKS deployment. The private API connectivity block represents the supported cluster API access model, which can vary by AKS configuration. DNS, management and application traffic are separate logical flows; the animation does not imply that every connection traverses every component.
06.2 / Private API Server Access

Understanding Private AKS Control-Plane Connectivity

Kubernetes API server connectivity is required for cluster management and node-to-control-plane communication. In private AKS deployments, the configured private access mechanism must be reachable from authorised management networks.

MANAGEMENT PATH 01
Cluster Nodes

Worker nodes need supported network and DNS connectivity to the managed API server.

MANAGEMENT PATH 02
Administrators

Management clients require authorised private connectivity and correct API hostname resolution.

MANAGEMENT PATH 03
Automation Systems

CI/CD runners, GitOps components and management tooling must be placed on reachable, authorised network paths.

Design distinction: Private AKS, API server VNet integration and other supported API exposure options have different implementation details and constraints. Confirm the selected cluster configuration rather than assuming every private AKS cluster uses an identical private endpoint architecture.

06.3 / Azure Private DNS

Why DNS Is Critical to Private AKS Connectivity

Private API connectivity depends on resolving the API server hostname to the appropriate private address. Azure Private DNS zones, VNet links and DNS forwarding infrastructure may be involved depending on the AKS configuration.

In enterprise environments using custom DNS servers, the resolver architecture must forward relevant private-zone queries to a resolver capable of resolving the Azure private DNS records.

DNS ComponentResponsibilityFailure Risk
AKS API hostnameIdentifies the managed Kubernetes API endpoint.Incorrect or missing resolution prevents API access.
Private DNS zoneStores applicable private endpoint records.Missing records or links can break name resolution.
VNet DNS configurationDetermines which resolvers Azure workloads query.Incorrect custom DNS settings affect cluster dependencies.
DNS forwardingConnects enterprise DNS to Azure private resolution.Incorrect forwarding targets cause lookup failures.
DNS cacheTemporarily stores resolved responses.Stale results may complicate endpoint changes.
06.4 / Azure Private Link

Connecting AKS Applications to Azure PaaS Privately

Azure Private Link allows supported services to be accessed through private endpoints associated with Azure virtual networks.

A private endpoint has a private IP address in a VNet subnet. Applications use the service's supported DNS naming model to resolve traffic toward the private endpoint.

CONCEPTUAL PRIVATE ENDPOINT FLOW AKS Application Pod | v DNS Lookup | v Private DNS Resolution | v Private Endpoint IP | v Azure VNet Routing | v Supported Azure PaaS Service VALIDATE: - Private endpoint approval/state - Private DNS zone integration - VNet links / forwarding - NSG and routing configuration - Service-side access controls - Source NAT and return path

Private endpoint connectivity and service authorisation are separate requirements. Reaching the endpoint IP does not automatically grant access to the underlying service.

06.5 / Hub-and-Spoke DNS Architecture

Resolving Private Azure Names From Enterprise Networks

Large Azure environments often centralise DNS resolution in a connectivity hub. Spoke VNets and enterprise networks use forwarding rules or custom resolvers to resolve private Azure service names.

DNS LAYER 01
AKS Workload DNS

Application pods commonly query cluster DNS, which resolves Kubernetes names and forwards other queries according to configuration.

DNS LAYER 02
Azure Private Resolution

Azure Private DNS zones and appropriate resolver connectivity provide private address records.

DNS LAYER 03
Enterprise DNS

On-premises DNS servers can forward relevant private-zone queries through supported hybrid DNS architecture.

DNS engineering principle: DNS resolution and IP connectivity must both succeed. A correct private IP response does not prove the network path is reachable, and a reachable endpoint IP does not prove the hostname resolves correctly.

06.6 / Hybrid Management Access

ExpressRoute, VPN and Private AKS Administration

Enterprise administrators may manage private AKS clusters from connected corporate networks through supported ExpressRoute or VPN connectivity.

The management path must provide correct API hostname resolution, routing to the private API endpoint and any required security permissions.

RequirementEngineering Validation
Hybrid connectivityConfirm ExpressRoute or VPN connection state and routing.
Private DNSResolve the AKS API hostname from the management client.
Route reachabilityValidate the path to the API server's private address.
Security filteringCheck applicable firewalls, NSGs and management access controls.
AuthenticationConfirm authorised Kubernetes and Azure identity access.
Interactive Private AKS Connectivity Explorer

Trace Private API, DNS and Application Paths

Select a scenario to inspect the logical connectivity requirements.

Private AKS API Server Connectivity

An authorised management client resolves the AKS API hostname and connects through the supported private API access path. DNS, routing, security controls and Kubernetes authentication must all succeed.

Management Client → Private DNS → Private API Access → AKS API Server
06.7 / Operational Validation

Inspecting Private AKS and DNS Configuration

# Inspect AKS private-cluster settings az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query "apiServerAccessProfile" # Inspect the API server hostname az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query "privateFqdn" # Inspect Azure Private DNS zones az network private-dns zone list \ --resource-group <resource-group> \ --output table # Inspect private endpoints az network private-endpoint list \ --resource-group <resource-group> \ --output table # Inspect VNet DNS settings az network vnet show \ --resource-group <resource-group> \ --name <vnet-name> \ --query "dhcpOptions" # Resolve API hostname from a # connected management host nslookup <aks-private-fqdn> # Test TCP reachability where supported curl -vk \ https://<aks-private-fqdn>:443 # Inspect Kubernetes cluster state kubectl get nodes -o wide

The availability of privateFqdn and the relevant DNS resources depends on the AKS API connectivity model. Use the actual cluster configuration to determine which records and endpoints should exist.

06.8 / Production Troubleshooting

Diagnosing Private AKS Connectivity Failures

SymptomLikely Investigation AreaRecommended Check
kubectl cannot resolve API hostnamePrivate DNS or forwarding configuration.Inspect resolver path and applicable private DNS records.
Hostname resolves but connection times outRouting, firewall or private endpoint reachability.Trace TCP connectivity and Azure network path.
On-premises access failsHybrid routing or DNS forwarding.Validate ExpressRoute/VPN and enterprise DNS.
AKS pods cannot reach private PaaSPrivate endpoint DNS, policy or routing.Test name resolution and connectivity from a workload.
API responds with unauthorised errorAuthentication or RBAC.Validate identity and Kubernetes permissions.
Only one VNet failsMissing DNS link, forwarding or routing.Compare VNet DNS and connectivity configuration.

Production troubleshooting sequence: Validate hostname resolution first, then IP reachability, then TLS/API response and finally authentication and authorisation. These are distinct failure domains and should not be diagnosed as a single private-cluster problem.

06.9 / Engineering Summary

The Private AKS Networking Mental Model

Private AKS controls API server connectivity. Azure Private DNS enables private name resolution. Azure Private Link connects supported services through private endpoints. ExpressRoute and VPN extend authorised enterprise connectivity.

Reliable private AKS deployments depend on consistent DNS architecture, correct routing, explicit access controls and a clear distinction between control-plane management and application traffic.

Network Insight · Engineering Guide 22 · Lab 06

Private AKS Connectivity Explorer

Investigate private AKS API server connectivity, Azure Private DNS, enterprise DNS forwarding, hub-spoke networking and hybrid access through ExpressRoute or VPN.

Private AKS Enterprise Connectivity Architecture

Select a component to inspect its role. Trace a management client through DNS resolution, private routing and API server connectivity.

PRIVATE AKS · ENTERPRISE API CONNECTIVITY ENTERPRISE NETWORK AZURE HUB VNET AKS SPOKE / PRIVATE API ADMIN CLIENT kubectl / Azure CLI EXPRESSROUTE / VPN Private Hybrid Path Enterprise Routing DNS RESOLVER Conditional Forwarding Hybrid Name Resolution HUB-SPOKE PEERING Private VNet Routing Gateway / Route Settings PRIVATE DNS ZONE Private API FQDN Private IP Resolution PRIVATE AKS API TCP 443 / TLS Private Endpoint Path DNS LOOKUP + PRIVATE TCP CONNECTIVITY API FQDN → PRIVATE IP → TCP 443 → TLS → AUTHENTICATION
RESOLVED DNS status
READY Private routing
443 API TCP port
READY Connectivity verdict
Select an architecture component to inspect its private connectivity responsibilities.

Private AKS Engineering Console

Private Connectivity State
Private DNS record exists
DNS forwarding path operational
VNet peering / routing available
Hybrid connectivity available
Network controls permit TCP 443
Select a management client and trace the private API connection.

Private DNS and Connectivity State

DependencyExpected BehaviorState

Private API Connection Pipeline

1. Management client requests AKS API FQDN
2. DNS forwarding and private-zone lookup
3. Private API address returned
4. Hybrid / VNet routing evaluated
5. TCP 443 and TLS connectivity checked
6. Kubernetes API authentication begins

Private AKS Diagnostic Terminal

[NETWORK INSIGHT · AKS LAB 06] Cluster: aks-production API access: Private Transport: ExpressRoute DNS: Private Zone Awaiting private API connectivity trace.

Root Cause and Corrective Action

Private DNS: The management client must resolve the cluster API FQDN to a reachable private address.
Private Routing: The client requires a valid route to the private API endpoint.
API Connectivity: Network policy, TCP 443 and TLS must succeed before Kubernetes authentication.

Production Engineering Notes

Private AKS clusters use private connectivity for Kubernetes API server access. The exact private endpoint architecture, DNS zone names and access options depend on the cluster configuration and supported AKS features.

An enterprise management workstation requires both correct DNS resolution and private network reachability. Successful name resolution does not guarantee that TCP 443 is reachable.

For hybrid access, enterprise DNS may use conditional forwarding through supported Azure DNS resolver or forwarding infrastructure. Private DNS zone links and resolver rules must match the deployed topology.

VNet peering alone does not provide universal transitive routing. Hub-spoke and hybrid connectivity must be configured with appropriate routes, gateway transit and security controls.

Private AKS API access is distinct from private access to Azure PaaS services through customer Private Endpoints. Both may involve Private Link and DNS, but their resource models and configuration requirements differ.

Educational simulation. The example private API address and DNS behavior are illustrative. The widget does not connect to Azure or execute real kubectl, DNS or TCP diagnostics.

Engineering Guide 22 · Editorial Section 07

AKS Cilium eBPF Datapath and Observability

Cilium uses extended Berkeley Packet Filter technology to implement supported networking and security functions inside the Linux kernel, helping AKS process workload traffic efficiently and enforce network policy.

In Azure CNI Powered by Cilium, the eBPF datapath participates in packet forwarding, Kubernetes Service handling and supported policy enforcement. The exact feature set and implementation depend on the AKS release and selected networking mode.

Understanding where packets enter the datapath, how forwarding decisions are made and where drops occur is fundamental to production AKS troubleshooting.

07.1 / eBPF Datapath Architecture

How Cilium Processes Kubernetes Network Traffic

eBPF programs execute at supported Linux kernel hooks and can use kernel maps to store networking state such as endpoint information, policy data and Service translation entries.

This architecture enables selected networking operations to occur within the kernel datapath rather than requiring every forwarding decision to pass through a separate user-space process.

CILIUM eBPF — LOGICAL PACKET PROCESSING AND OBSERVABILITY AKS WORKER NODE — LINUX NETWORK DATAPATH Source Pod Application Packet Linux eBPF Hook Supported Datapath Entry Policy Evaluation Identity / Policy State Forwarding Next Hop / Endpoint eBPF Maps Endpoint · Service · Policy State Flow Observability Drops · Counters · Diagnostics Destination Pod / Network
Figure 7 — Conceptual Cilium datapath model. The diagram represents logical processing functions rather than a fixed sequence of Linux hooks. Actual execution order, map usage, policy checks and observability mechanisms vary by datapath and AKS configuration.
07.2 / Linux Kernel Hooks

Where eBPF Programs Interact With Network Packets

Linux supports several eBPF attachment mechanisms. Cilium can use supported hooks to implement networking and security behaviour.

MechanismGeneral FunctionEngineering Relevance
Traffic Control (TC)Packet processing at supported ingress and egress points.Forwarding, policy enforcement and packet manipulation.
XDPEarly packet processing on supported network drivers.High-performance packet handling in compatible environments.
Socket-level hooksProcessing associated with supported socket operations.Can support selected Service and connection handling.
eBPF mapsKernel-resident data structures.Networking state, endpoint data and policy lookups.
Perf / ring-buffer mechanismsExport supported events from kernel processing.Diagnostics and telemetry pipelines.

The presence of a Linux eBPF capability does not guarantee that AKS exposes or enables it. Managed platform support and the deployed Cilium implementation determine the usable feature set.

07.3 / Endpoint Identity and Policy

How Cilium Associates Workloads With Security Decisions

Cilium can associate workloads with security identities derived from relevant labels and policy state. These identities help the datapath evaluate communication permissions.

In managed AKS, engineers should primarily use supported Kubernetes policy interfaces and confirm the available Cilium policy features.

IDENTITY
Workload Labels

Kubernetes labels describe workloads and provide selection criteria for network policies.

POLICY
Allowed Communication

Supported policy state determines which selected traffic is permitted.

DATAPATH
Enforcement Decision

Kernel networking functions enforce supported allow and deny decisions.

Engineering distinction: Kubernetes network policy is declarative configuration. Cilium eBPF is one supported implementation mechanism. The policy object does not itself execute packet forwarding.

07.4 / Service Load Balancing

How Cilium Can Implement Kubernetes Service Forwarding

Kubernetes Services provide stable virtual addresses for dynamic backend endpoints. Supported Cilium datapath functionality can perform Service lookup and backend selection using kernel-resident networking state.

Service forwarding must account for backend readiness, connection state, traffic policies and the selected AKS networking mode.

CONCEPTUAL SERVICE PROCESSING Application Packet | v Destination: Service VIP | v Supported Service Lookup | v Eligible Backend Selection | v Forwarding / Address Translation | v Destination Pod OPERATIONAL CHECKS: - Service selector - EndpointSlice state - Backend readiness - Service traffic policy - Network policy - Node reachability - Cilium datapath support
07.5 / Observability Architecture

Understanding Flow Visibility and Packet Drops

Datapath observability helps engineers determine whether traffic was forwarded, denied by policy, dropped by the network stack or affected by another connectivity problem.

Upstream Cilium provides several observability capabilities, but availability in AKS depends on the supported managed networking integration and the diagnostic tools exposed to operators.

SignalWhat It Can RevealOperational Use
Policy verdictWhether supported policy enforcement permitted or denied traffic.Investigate segmentation failures.
Packet drop informationPotential datapath or forwarding failures.Identify where packets stop progressing.
Service backend stateEligible endpoints for a Kubernetes Service.Diagnose application connectivity.
Node network countersInterface errors, drops and traffic volume.Investigate performance and transport issues.
Application telemetryLatency, errors and connection behaviour.Correlate network and application symptoms.

Observability caution: Do not assume that Hubble, upstream Cilium CLI access or all Cilium flow-export capabilities are available in Azure CNI Powered by Cilium. Verify supported AKS monitoring and diagnostics before designing an observability workflow.

07.6 / Performance Engineering

Latency, Throughput and Datapath Efficiency

eBPF can support efficient packet processing, but application performance still depends on node resources, network bandwidth, packet size, transport behaviour, routing and workload characteristics.

PERFORMANCE 01
CPU and Packet Processing

High packet rates can increase CPU pressure even when datapath processing is efficient.

PERFORMANCE 02
MTU and Encapsulation

Overlay transport can introduce packet-size considerations that require correct MTU planning.

PERFORMANCE 03
Connection Behaviour

Connection churn, backend changes and transport retries can affect application latency.

Engineers should benchmark representative workloads and inspect the complete network path rather than attributing every performance improvement or regression directly to eBPF.

Interactive Cilium Datapath Explorer

Inspect Six Packet-Processing Scenarios

Select a datapath function to understand its role and the corresponding troubleshooting evidence.

Packet Entry Into the Node Datapath

A packet enters the configured Linux networking path through a pod interface or node network interface. Supported eBPF programs can process traffic at the relevant hooks, depending on the AKS datapath implementation.

Pod / Node Interface → Linux Networking Hook → Datapath Processing
07.7 / Operational Diagnostics

Investigating AKS Cilium Networking State

# Inspect AKS network profile az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query networkProfile # Inspect networking system pods kubectl get pods -n kube-system # Inspect node and pod placement kubectl get nodes -o wide kubectl get pods -A -o wide # Inspect Kubernetes Services kubectl get services -A # Inspect EndpointSlices kubectl get endpointslices -A # Inspect NetworkPolicy resources kubectl get networkpolicies -A # Inspect node resource conditions kubectl describe node <node-name> # Inspect pod events kubectl describe pod \ <pod-name> \ -n <namespace> # On an authorised node shell, # where supported: ip -s link ip route ss -s # Use AKS-supported networking # diagnostics and monitoring tools # for deeper datapath inspection.

Avoid making direct modifications to managed Cilium components or kernel maps. Such changes may be unsupported and can disrupt cluster networking.

07.8 / Production Failure Analysis

Diagnosing Cilium and eBPF Networking Symptoms

SymptomPossible Failure DomainInvestigation
Traffic fails after policy deploymentPolicy selector or enforcement state.Inspect applicable policies and endpoint labels.
Service IP unreachableService forwarding or backend state.Inspect Service and EndpointSlice resources.
Cross-node traffic failsNode connectivity, overlay or datapath issue.Compare same-node and cross-node tests.
Intermittent latency spikesCPU pressure, transport retries or backend churn.Correlate node and application telemetry.
Large packets failMTU or path-MTU problem.Inspect interface MTUs and transport behaviour.
Observability data missingUnsupported feature or telemetry configuration.Verify AKS support and monitoring setup.

Production investigation principle: Identify the failing layer before changing networking configuration. Kubernetes policy, Service state, CNI forwarding, Azure routing and application behaviour must be examined as separate but connected failure domains.

07.9 / Engineering Summary

The AKS Cilium Datapath Mental Model

Cilium uses supported eBPF mechanisms to implement selected packet-processing, Service forwarding and network-policy capabilities. Linux kernel state supports datapath decisions, while Kubernetes provides workload and policy configuration.

Reliable AKS observability requires correlating datapath signals with Kubernetes state, Azure network infrastructure and application telemetry.

Network Insight · Engineering Guide 22 · Lab 07

AKS Cilium eBPF Packet Flow Explorer

Explore how a supported Cilium datapath processes AKS pod traffic, performs policy evaluation, handles Service forwarding and exposes flow telemetry for troubleshooting.

Cilium eBPF Packet Processing Architecture

Select a datapath component to inspect its responsibility. Run the simulator to trace packet decisions and inspect flow events.

AKS CILIUM · eBPF DATAPATH PROCESSING AKS WORKER NODE / LINUX DATAPATH SOURCE POD 10.244.0.10 Endpoint Identity eBPF DATAPATH Endpoint / TC Hooks Packet Processing DESTINATION POD 10.244.1.20 Backend Endpoint IDENTITY LOOKUP Endpoint Labels Security Identity POLICY ENGINE L3 / L4 Decision Allow / Drop SERVICE LOOKUP VIP → Backend eBPF Maps IDENTITY → POLICY → SERVICE → FORWARDING → TELEMETRY
READY Packet verdict
ALLOW Policy decision
— Modeled latency
ENABLED Flow telemetry
Select a datapath component to inspect its processing responsibility.

Cilium Datapath Engineering Console

eBPF Datapath State
Endpoint identity available
Service/backend map healthy
Forwarding route available
Flow telemetry enabled
Configure a packet flow and run the eBPF trace.

eBPF Processing Decision Matrix

StageDatapath ResponsibilityResult

Packet Processing Pipeline

1. Source pod packet enters datapath
2. Endpoint identity lookup
3. Workload policy evaluation
4. Service/backend lookup where required
5. Forwarding decision
6. Flow telemetry observation

Cilium Diagnostic Terminal

[NETWORK INSIGHT · AKS LAB 07] Datapath: Cilium eBPF Source: 10.244.0.10 Destination: 10.244.1.20 Policy mode: Least Privilege Awaiting packet trace.

Flow Observability and Root Cause

Flow visibility: Supported Cilium telemetry can expose forwarding and policy-related events.
Policy enforcement: Endpoint identities and configured policy rules influence traffic decisions.
Service forwarding: eBPF maps can support Service translation in applicable datapaths.

Production Engineering Notes

Cilium uses eBPF programs attached at supported Linux networking hooks. The precise processing order, available hooks and Service implementation depend on the Cilium version, AKS configuration and enabled datapath features.

Cilium endpoint identities allow policies to express workload-level security intent. Kubernetes NetworkPolicy remains an additive allow-policy model; Cilium-specific policy capabilities require checking managed AKS support.

Service traffic may use eBPF-based translation and backend selection. Packet transformations depend on the Service type, datapath mode and cluster networking configuration.

Hubble-style observability can expose flow verdicts, identities and traffic metadata in supported Cilium deployments. Observability availability and the supported collection interfaces should be verified for the particular AKS environment.

Educational simulation. Flow events, latency figures, map lookups and processing stages are illustrative. The widget does not query Cilium, Hubble, Azure or a live Kubernetes cluster.

Engineering Guide 22 · Editorial Section 08

AKS High Availability and Multi-Region Networking

Production AKS environments must remain available when pods fail, nodes become unavailable, availability zones experience disruption or an entire Azure region becomes inaccessible.

High availability is not provided by Kubernetes scheduling alone. It requires coordinated workload placement, resilient Azure networking, healthy load-balancer backends, regional traffic management and dependable application data services.

The engineering objective is to minimise failure impact while maintaining predictable traffic paths, recoverability and operational control.

08.1 / Multi-Region Architecture

Designing Resilient AKS Application Delivery

A multi-region AKS architecture typically uses separate clusters in different Azure regions. Each cluster has its own worker nodes, networking resources, application deployments and operational dependencies.

A global traffic distribution service can direct clients toward healthy regional application endpoints. Azure Front Door is one option for supported HTTP and HTTPS workloads, while Azure Traffic Manager provides DNS-based traffic distribution.

AKS MULTI-REGION HIGH AVAILABILITY ARCHITECTURE Global Clients HTTPS Application Traffic Azure Front Door / Global Routing Health Probes · Regional Origin Selection AZURE REGION A — PRIMARY AKS AZURE REGION B — SECONDARY AKS Regional Ingress / Load Balancer Healthy Regional Endpoint Regional Ingress / Load Balancer Standby or Active Endpoint Availability Zone 1 Node Pool · Pods Availability Zone 2 Node Pool · Pods Availability Zone 1 Node Pool · Pods Availability Zone 2 Node Pool · Pods
Figure 8 — Conceptual multi-region AKS application architecture. Azure Front Door is shown as an example for HTTP(S) traffic; Traffic Manager is a DNS-based alternative with different behaviour. Regional clusters, ingress endpoints and application state must be deployed and configured separately. The diagram does not imply automatic state replication or guaranteed zero-downtime failover.
08.2 / Availability Zones

Distributing Workloads Across Independent Failure Domains

Azure availability zones provide physically separate infrastructure locations within supported regions. AKS can use supported zone-aware node pool configurations to distribute worker nodes across zones.

Kubernetes scheduling constraints help distribute application replicas so that a single node or zone failure does not remove every available instance.

FAILURE DOMAIN 01
Pod Failure

Multiple replicas and readiness checks help maintain service availability when individual application instances fail.

FAILURE DOMAIN 02
Node Failure

Replica distribution across nodes reduces dependence on a single worker host.

FAILURE DOMAIN 03
Zone Failure

Zone-aware node placement and application scheduling can reduce the impact of an availability-zone outage.

Availability principle: A zone-spanning node pool does not automatically guarantee that every workload is distributed across zones. Pod topology spread constraints, affinity rules, replica counts and resource capacity must also be designed correctly.

08.3 / Kubernetes Scheduling Resilience

Topology Spread, Pod Disruption Budgets and Replicas

Kubernetes provides several workload-level mechanisms that help maintain availability during failures and planned maintenance.

MechanismPurposeEngineering Consideration
Deployment replicasMaintain multiple application instances.Replica count must match failure-domain requirements.
Topology spread constraintsDistribute pods across selected topology domains.Requires suitable node labels and scheduling capacity.
Pod anti-affinityDiscourage or prevent co-location.Strict rules can block scheduling when capacity is limited.
PodDisruptionBudgetLimit selected voluntary disruptions.Does not prevent involuntary failures such as node outages.
Readiness probesIndicate whether pods should receive Service traffic.Incorrect probes can remove healthy endpoints or retain unhealthy ones.
Cluster autoscalerAdjust eligible node pool capacity.Scaling is not instantaneous and depends on available infrastructure.
# Example topology spread configuration # Place inside a Deployment pod spec. topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: frontend # Example PodDisruptionBudget apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: frontend-pdb namespace: production spec: minAvailable: 2 selector: matchLabels: app: frontend

These examples require matching workload labels and sufficient node capacity. A PodDisruptionBudget protects against supported voluntary disruptions, not unexpected infrastructure loss.

08.4 / Regional Load Balancing

Keeping Application Traffic Away From Unhealthy Backends

Regional application delivery commonly uses Azure Standard Load Balancer and an ingress or Gateway implementation. Backend health, Kubernetes readiness and node availability all influence whether traffic reaches a functioning application instance.

Azure Load Balancer operates at Layer 4. Application-level health decisions may require an ingress controller or a higher-layer traffic distribution service.

REGIONAL LAYER
Azure Load Balancer

Distributes supported network connections to healthy backend targets.

KUBERNETES LAYER
Service Endpoints

Uses eligible application endpoints derived from Kubernetes workload state.

APPLICATION LAYER
Ingress / Gateway

Implements supported application routing, TLS handling and HTTP traffic management.

08.5 / Multi-Region Traffic Routing

Azure Front Door and Azure Traffic Manager

CapabilityAzure Front DoorAzure Traffic Manager
Traffic modelGlobal HTTP(S) reverse-proxy service.DNS-based endpoint selection.
Routing decisionsSupported origin health, routing and load-balancing configuration.Configured DNS traffic-routing method and endpoint monitoring.
Failover behaviourNew requests can be directed to healthy origins according to configuration.DNS answers change; client caching and TTL influence transition.
Application protocolsPrimarily HTTP and HTTPS workloads.DNS-based distribution for services reachable through configured endpoints.
Operational focusOrigin health, routing rules, TLS and application availability.Endpoint health, DNS behaviour and client resolution.

Neither service automatically synchronises application data, database state or Kubernetes configuration between regional clusters.

Failover engineering rule: A global traffic service can redirect traffic only to an endpoint that is configured, reachable and sufficiently healthy. The secondary AKS deployment must already have the necessary application capacity and dependencies.

08.6 / Disaster Recovery Architecture

RTO, RPO and Application State

Multi-region disaster recovery must consider more than restoring Kubernetes pods. Applications may depend on databases, storage, messaging systems, identity providers and external APIs.

RECOVERY METRIC
Recovery Time Objective

RTO defines the target maximum time required to restore service after a disruption.

RECOVERY METRIC
Recovery Point Objective

RPO defines the acceptable amount of data loss measured in time.

RECOVERY MODEL
Active / Passive or Active / Active

Deployment models differ in cost, operational complexity, capacity and consistency requirements.

Data replication design, application consistency and recovery procedures must be tested independently from network failover.

Interactive AKS Resilience Explorer

Investigate Six High-Availability Failure Scenarios

Select a scenario to inspect the expected recovery behaviour and its dependencies.

Application Pod Failure

When a pod fails, Kubernetes controllers can create a replacement replica. Service endpoint state and readiness checks help prevent new traffic from being sent to an unavailable application instance.

Pod Failure → Endpoint Update → Healthy Replicas → Replacement Pod
08.7 / Operational Validation

Inspecting AKS Availability and Regional Networking

# Inspect AKS node pools az aks nodepool list \ --resource-group <resource-group> \ --cluster-name <cluster-name> \ --output table # Inspect Kubernetes node topology kubectl get nodes \ -L topology.kubernetes.io/zone # Inspect pod placement kubectl get pods -A -o wide # Inspect deployment availability kubectl get deployments -A # Inspect disruption budgets kubectl get pdb -A # Inspect Services and endpoints kubectl get services -A kubectl get endpointslices -A # Inspect application ingress kubectl get ingress -A # Inspect cluster events kubectl get events -A \ --sort-by=.lastTimestamp # Inspect Azure Front Door resources # using the applicable Azure CLI # extension and resource configuration. # Inspect regional dependencies # and validate failover procedures.

A production validation plan should include controlled node disruption, zone-aware scheduling checks, regional health-probe testing and application-level recovery verification.

08.8 / Production Failure Analysis

Diagnosing Availability and Failover Problems

SymptomPossible CauseInvestigation
All replicas fail togetherShared node or zone placement.Inspect pod scheduling and topology distribution.
Pods remain Pending after failureInsufficient capacity or restrictive scheduling.Inspect events, node pools and autoscaler state.
Traffic reaches unhealthy regionHealth-probe or global routing configuration.Inspect origin health and traffic distribution rules.
Regional failover is slowProbe intervals, DNS caching or application startup.Measure each stage of recovery.
Application returns errors after failoverMissing regional dependency or inconsistent data.Validate storage, databases and service configuration.
Zone outage removes all capacityInsufficient cross-zone node or replica distribution.Inspect zone placement and resource quotas.

Production resilience principle: High availability must be measured through application outcomes, not simply by counting healthy Kubernetes nodes. An available cluster can still host an unavailable application if networking, DNS or data dependencies fail.

08.9 / Engineering Summary

The AKS High-Availability Mental Model

Availability zones protect against selected infrastructure failures within a region. Kubernetes scheduling distributes application replicas. Azure load balancing manages regional traffic. Global routing services can redirect supported workloads between healthy regional endpoints.

A resilient production design must also account for DNS, identity, application state, capacity, recovery objectives and regularly tested failover procedures.

Network Insight · Engineering Guide 22 · Lab 08

AKS High Availability and Failure Simulator

Explore application resilience across availability zones, node pools, Azure Load Balancer health, replica placement and multi-region disaster recovery.

Multi-Zone and Multi-Region AKS Architecture

Select a node, zone or region to inspect its resilience role. Run a failure scenario to evaluate remaining capacity and availability.

AKS HIGH AVAILABILITY · MULTI-ZONE / MULTI-REGION GLOBAL TRAFFIC ROUTING Health-Based Regional Selection PRIMARY AZURE REGION SECONDARY REGION REGIONAL LOAD BALANCER Health Probe / Backend Pool ZONE 1 Worker Node A Pod Replica A Capacity: 2 Pods ZONE 2 Worker Node B Pod Replica B Capacity: 2 Pods ZONE 3 Worker Node C Pod Replica C Capacity: 2 Pods SECONDARY AKS Independent Cluster Standby Capacity Regional Failover
3 / 3 Ready replicas
6 Available pod slots
PRIMARY Serving region
READY Availability verdict
Select a zone, Load Balancer or region to inspect its high-availability role.

AKS Resilience Engineering Console

Infrastructure and Recovery State
Availability Zone 1 healthy
Availability Zone 2 healthy
Availability Zone 3 healthy
Regional Load Balancer healthy
Secondary AKS region available
Rescheduling / recovery enabled
Configure an infrastructure failure and evaluate resilience.

AKS Replica and Capacity State

LocationCapacityHealthRole

Failure and Recovery Pipeline

1. Detect infrastructure or application failure
2. Evaluate surviving availability zones
3. Calculate remaining workload capacity
4. Reconcile desired replicas
5. Evaluate regional traffic health
6. Determine failover or degraded availability

AKS Incident Timeline

[NETWORK INSIGHT · AKS LAB 08] Primary region: Healthy Availability zones: 3 Desired replicas: 3 Secondary region: Available Awaiting failure simulation.

Resilience Root Cause Analysis

Availability zones: Distribute nodes and application replicas across independent zone failure domains.
Recovery capacity: Surviving nodes must have sufficient resources to host replacement replicas.
Regional recovery: Multi-region continuity requires independent application deployments, traffic routing and data recovery design.

Production Engineering Notes

Availability zones provide distinct infrastructure failure domains within supported Azure regions. AKS node pools and workload scheduling must be configured appropriately to achieve the intended zone distribution.

Kubernetes Deployments reconcile desired replica counts, but replacement pods require sufficient available resources, compatible scheduling constraints, working image pulls and healthy nodes. Recovery is not instantaneous.

Pod topology spread constraints, pod disruption budgets, node pool configuration and application readiness all influence availability. Pod disruption budgets primarily control voluntary disruptions and do not prevent infrastructure failures.

Azure Load Balancer health probes and Service endpoint readiness influence traffic delivery. A healthy Kubernetes control plane does not guarantee that an application is available.

Multi-region AKS architectures require separate cluster deployments and an appropriate global traffic-routing mechanism, such as Azure Front Door or Traffic Manager where suitable. Data replication, DNS behavior, identity and application dependencies must also be included in the disaster recovery plan.

Educational simulation. Each healthy primary zone provides two illustrative pod slots. Replica reconciliation, health detection and regional failover are simplified models, not real Azure availability guarantees or measured RTO/RPO.

Engineering Guide 22 · Editorial Section 09 · Final Section

AKS Production Troubleshooting and Performance Engineering

Troubleshooting Azure Kubernetes Service requires a structured understanding of Kubernetes workload state, Cilium networking, Azure routing, DNS, load balancing and the underlying Linux network datapath.

A failed application connection may originate from an incorrect Kubernetes Service selector, a network policy drop, a DNS resolution failure, a missing Azure route, SNAT port exhaustion, an unhealthy load-balancer backend or an application that is not ready to accept traffic.

The objective of production network troubleshooting is to identify the precise failure domain, collect evidence and restore service without introducing unnecessary changes to unrelated networking components.

09.1 / Production Network Architecture

Tracing an AKS Application Connection End to End

AKS networking spans several independently managed layers. Kubernetes controls workloads and Services, the CNI provides pod networking, Linux processes packets, and Azure provides virtual networks, routing, load balancing and security infrastructure.

Engineers must determine whether a failure occurs before a packet reaches the cluster, inside the Kubernetes datapath or after the packet leaves the node.

AKS PRODUCTION TROUBLESHOOTING — END-TO-END PACKET INVESTIGATION External Client Application Request DNS Resolution Public / Private DNS Azure Load Balancer Frontend / Health Probe Failure Signals Timeout / 5xx / Drop AKS CLUSTER — SERVICE AND DATAPATH Kubernetes Service VIP / EndpointSlices Cilium eBPF Policy / Forwarding Application Pod Ready / Listening Drop / Error Policy / MTU Azure Routing / UDR Next Hop / VNet Azure Firewall / NAT SNAT / Filtering External Dependency API / Database / Service
Figure 9 — Conceptual AKS investigation topology. The arrows illustrate possible traffic and diagnostic relationships, not a mandatory packet sequence. DNS lookup, inbound delivery and outbound dependency access are distinct operations. Cilium, Azure routing and application behaviour must be investigated in the context of the actual deployment.
09.2 / Structured Incident Investigation

The Five-Stage AKS Troubleshooting Method

An effective incident workflow begins by defining the observed failure and then testing progressively narrower network boundaries.

STAGE 01 Establish the Symptom
STAGE 02 Validate DNS and Endpoints
STAGE 03 Inspect the Datapath
STAGE 04 Verify Azure Networking
STAGE 05 Confirm Recovery
INITIAL TRIAGE
Define the Failure Boundary

Determine whether the problem affects one pod, one namespace, one node, one subnet, one cluster or an entire region.

EVIDENCE COLLECTION
Compare Working and Failing Paths

Test same-node, cross-node, internal Service and external connectivity to isolate the failing network layer.

RECOVERY VALIDATION
Measure the Actual Outcome

Confirm application availability, request success rates, latency and backend health after remediation.

09.3 / DNS Troubleshooting

Diagnosing Kubernetes and Azure DNS Failures

DNS problems can appear as application timeouts, failed API calls or unreachable Kubernetes Services even when the underlying IP network is healthy.

AKS DNS investigations should distinguish cluster Service discovery from Azure private DNS resolution and external hostname lookup.

DNS Failure Likely Investigation Area Diagnostic Action
Service hostname does not resolve CoreDNS, Service name or namespace. Inspect Service objects and cluster DNS.
Private endpoint resolves publicly Private DNS zone integration or forwarding. Trace resolver configuration and DNS answers.
DNS fails after network policy deployment Missing egress permissions to DNS resolvers. Inspect policy selectors and UDP/TCP port 53.
Intermittent lookup latency Resolver load, forwarding or upstream dependency. Correlate DNS latency and resolver metrics.
# Inspect Kubernetes DNS components kubectl get pods -n kube-system # Inspect DNS Service kubectl get service kube-dns \ -n kube-system # Inspect DNS endpoints kubectl get endpointslices \ -n kube-system # Inspect DNS-related events kubectl get events -n kube-system \ --sort-by=.lastTimestamp # From a diagnostic pod with nslookup: nslookup kubernetes.default.svc.cluster.local # Resolve an Azure private service name nslookup <private-service-fqdn>

Diagnostic principle: DNS resolution should be tested from the affected workload's network and resolver context. Successful resolution on an administrator's laptop does not prove that AKS pods use the same DNS path.

09.4 / SNAT and Outbound Connectivity

Understanding SNAT Port Exhaustion

Outbound AKS connections may require source network address translation when workloads communicate with external destinations.

Depending on the cluster egress design, SNAT can be provided through supported Azure load-balancing, NAT Gateway, firewall or other configured networking components.

SNAT exhaustion occurs when available translation resources are insufficient for the active connection workload. It can produce intermittent outbound connection failures even while internal pod networking continues to function.

CONNECTION PRESSURE
High Outbound Concurrency

Large numbers of simultaneous connections can consume available SNAT translation resources.

APPLICATION BEHAVIOUR
Connection Churn

Repeatedly opening short-lived connections instead of reusing them can increase port pressure.

EGRESS DESIGN
NAT Capacity Planning

NAT Gateway or other supported outbound architectures may provide more appropriate capacity for demanding workloads.

CONCEPTUAL SNAT INVESTIGATION Application Pod | v AKS Node / Egress Datapath | v Configured Outbound Component | +-- Azure Load Balancer SNAT | +-- NAT Gateway | +-- Azure Firewall / Other Egress | v External Destination INVESTIGATE: - Concurrent outbound connections - Source NAT capacity and metrics - Destination IP / port concentration - Connection reuse - TCP connection lifetime - Firewall and routing behaviour

The correct mitigation depends on the actual outbound configuration. Increasing application connection reuse, adjusting egress architecture or adding supported NAT capacity may be appropriate after confirming the bottleneck.

09.5 / MTU and Overlay Networking

Diagnosing Packet Size and Fragmentation Problems

Azure CNI Overlay introduces an additional network transport context between pods on different nodes. Effective packet-size limits must account for the selected encapsulation and underlying Azure network path.

MTU mismatches can cause failures that are difficult to recognise because small requests may succeed while larger transfers stall or time out.

Symptom Potential Cause Investigation
Small packets succeed; large packets fail Path-MTU discovery or encapsulation mismatch. Compare effective MTU along the path.
Cross-node traffic fails selectively Overlay transport or node networking. Compare same-node and cross-node connectivity.
TLS connections stall Possible packet-size or transport issue. Inspect retransmissions and packet-size behaviour.
Unexpected packet drops Datapath, interface or route-related drops. Correlate node counters and supported diagnostics.
# On an authorised Linux host or # supported diagnostic environment: # Inspect interface MTUs ip link show # Inspect route selection ip route get <destination-ip> # Inspect interface counters ip -s link # Example path-MTU investigation # from a compatible diagnostic host: ping -M do -s 1400 <destination-ip> # The test size must be adjusted # for protocol overhead and the # actual path. ICMP filtering can # also affect the result.

MTU engineering rule: Do not change production node or CNI MTU settings solely because a large-packet test fails. Confirm the supported AKS networking configuration, path-MTU behaviour and the precise transport failure first.

09.6 / Cilium Policy and Datapath Failures

Distinguishing Policy Drops From Routing Failures

Cilium policy enforcement and Linux packet forwarding can produce symptoms similar to Azure routing or firewall failures.

An effective investigation compares Kubernetes policy configuration, workload labels, Service state, node placement and available datapath diagnostics.

POLICY FAILURE
Unexpected Deny

Incorrect selectors, missing egress rules or default-deny policies may block required application communication.

SERVICE FAILURE
Missing Backend

Incorrect Service selectors or failed readiness checks can leave a Service without eligible endpoints.

NETWORK FAILURE
Cross-Node Reachability

Node routing, overlay connectivity or infrastructure filtering can disrupt traffic between workloads on different nodes.

# Inspect NetworkPolicy configuration kubectl get networkpolicies -A # Inspect a specific policy kubectl describe networkpolicy \ <policy-name> \ -n <namespace> # Inspect pod labels kubectl get pods -A --show-labels # Inspect Service endpoints kubectl get endpointslices -A # Inspect node placement kubectl get pods -A -o wide # Inspect AKS network profile az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query networkProfile

Cilium-specific CLI, flow monitoring and eBPF map inspection features must be verified against the supported managed AKS implementation. Do not assume unrestricted upstream Cilium access.

09.7 / Interactive Incident Investigation

AKS Production Troubleshooting Scenario Explorer

Select a production failure to inspect its likely failure domain, investigation sequence and expected diagnostic path.

DNS Resolution Failure

A workload cannot resolve a Kubernetes Service or external hostname. Investigate cluster DNS, resolver forwarding, applicable network policies and private DNS configuration.

Application → DNS Query → Resolver Failure → Connection Cannot Start

First check: compare DNS answers from the affected pod and a known-working diagnostic workload.

09.8 / Production Diagnostic Commands

Building an AKS Network Investigation Toolkit

Kubernetes and Azure CLI commands provide a baseline view of cluster networking, workload placement, application endpoints and Azure infrastructure.

# ================================================== # AKS CLUSTER HEALTH # ================================================== kubectl get nodes -o wide kubectl get pods -A -o wide kubectl get events -A \ --sort-by=.lastTimestamp # ================================================== # SERVICES AND ENDPOINTS # ================================================== kubectl get services -A kubectl get endpointslices -A kubectl get ingress -A # ================================================== # NETWORK POLICIES # ================================================== kubectl get networkpolicies -A # ================================================== # AKS NETWORK CONFIGURATION # ================================================== az aks show \ --resource-group <resource-group> \ --name <cluster-name> \ --query networkProfile # ================================================== # AZURE ROUTING # ================================================== az network route-table list \ --resource-group <resource-group> \ --output table # ================================================== # AZURE NETWORK SECURITY # ================================================== az network nsg list \ --resource-group <resource-group> \ --output table # ================================================== # AZURE LOAD BALANCERS # ================================================== az network lb list \ --resource-group <node-resource-group> \ --output table # ================================================== # PRIVATE ENDPOINTS # ================================================== az network private-endpoint list \ --resource-group <resource-group> \ --output table # ================================================== # APPLICATION VALIDATION # ================================================== kubectl describe pod \ <pod-name> \ -n <namespace> kubectl logs \ <pod-name> \ -n <namespace> # Run connectivity tests from an # authorised diagnostic pod using # tools included in that image.

Azure resources may exist in different resource groups, particularly the AKS-managed node resource group. Confirm the correct resource scope before investigating.

09.9 / Production Troubleshooting Decision Matrix

Mapping Symptoms to Network Failure Domains

Observed Symptom Likely Failure Domain First Diagnostic Action
Service hostname does not resolve DNS configuration or resolver reachability. Inspect cluster DNS and test resolution from the affected workload.
Service has no eligible endpoints Selector mismatch or pod readiness. Inspect EndpointSlices, pod labels and readiness.
External API calls fail intermittently SNAT, firewall, DNS or application dependency. Inspect egress path and connection metrics.
Only large transfers fail MTU, path-MTU discovery or transport behaviour. Compare packet sizes and retransmission behaviour.
Traffic fails after policy deployment Network policy enforcement. Inspect selected pods and applicable ingress/egress rules.
Cross-node traffic fails Overlay, CNI, node routing or infrastructure filtering. Compare same-node and cross-node tests.
Ingress returns HTTP 502 or 503 Backend readiness, routing or ingress controller. Inspect controller logs, Services and endpoints.
Private API server cannot be reached Private DNS, hybrid routing or access controls. Resolve API hostname and test the private network path.

Final engineering principle: Troubleshooting must be evidence-driven. Begin with the smallest reproducible failure, identify the affected network boundary, compare working and failing paths, and validate recovery using application-level measurements.

Engineering Guide 22 / Final Summary

The Complete AKS Networking Mental Model

AKS networking combines Kubernetes workload orchestration, Azure CNI pod connectivity, Cilium datapath processing, Kubernetes Services, Azure load balancing, private DNS, network policies, Azure routing and enterprise connectivity.

Each component operates at a distinct layer, but production application connectivity depends on all required layers functioning together.

The most effective AKS engineers do not troubleshoot Kubernetes, Cilium and Azure as isolated technologies. They understand how a packet moves across these systems, where each forwarding or security decision occurs, and how to verify the result.

That end-to-end understanding enables faster incident resolution, more reliable architecture design and better performance engineering across complex cloud-native environments.

Network Insight · Engineering Guide 22 · Lab 09

AKS Production Incident Investigation Console

Investigate production AKS failures using structured evidence collection, diagnostic commands, root-cause selection, remediation and application recovery verification.

Production AKS Incident Architecture

Inspect architecture components and identify the dependency responsible for the outage. Diagnostic evidence is revealed through the investigation console.

AKS PRODUCTION · END-TO-END INCIDENT INVESTIGATION APPLICATION DELIVERY / KUBERNETES DATAPATH CLIENT Application Request INGRESS HTTP / TLS Routing SERVICE VIP / EndpointSlice COREDNS Service Resolution DNS Query Path CILIUM eBPF Policy / Forwarding Flow Verdict BACKEND PODS Application Endpoints Readiness / Health AZURE EGRESS / ROUTING / SNAT UDR · Firewall · NAT Gateway · MTU DNS → INGRESS → SERVICE → POLICY → BACKEND → EGRESS
SEV-2 Incident severity
0 Diagnostic checks
100 Investigation score
OPEN Incident state
Select an architecture component to inspect its troubleshooting responsibilities.

Production Incident Investigation Console

Incident loaded. Begin evidence collection.

Diagnostic Evidence and Command Output

[NETWORK INSIGHT · AKS INCIDENT CONSOLE] Cluster: aks-production Namespace: production Status: Investigation required Select a diagnostic command.

Incident Investigation Timeline

Incident opened. Application connectivity is degraded.

Production Troubleshooting Checklist

Investigation StageEngineering ObjectiveStatus

Root Cause and Engineering Guidance

Collect evidence: Inspect Kubernetes and Azure networking dependencies before changing configuration.
Identify the first failing layer: DNS, ingress, Service, policy, routing, SNAT or workload health.
Verify recovery: A successful configuration change must be followed by an application test.

Production Engineering Notes

A reliable AKS troubleshooting workflow begins with a clear symptom, source, destination, protocol, port and time window. Engineers should establish whether the failure affects one pod, one node, one namespace, one Service or the entire cluster.

DNS troubleshooting should distinguish CoreDNS failures, upstream resolver issues, private DNS configuration and network connectivity to DNS servers. A failed name lookup is not proof that the application backend is unhealthy.

Service troubleshooting requires checking Service selectors, EndpointSlices, pod readiness, ingress rules and the selected Azure Load Balancer datapath. Network policy drops and routing failures can produce similar user-visible symptoms.

Outbound connection failures may involve SNAT capacity, Azure Firewall, NAT Gateway, UDR configuration, DNS or application connection behavior. The exact source of SNAT capacity depends on the deployed outbound architecture.

MTU failures are particularly important when overlay networking, encapsulation, hybrid links and firewalls interact. Symptoms may include small requests succeeding while larger transfers stall.

This is an educational incident simulator. All diagnostics, outputs, scores and remediation results are simulated. No commands are executed against a live AKS cluster or Azure subscription.

☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Matt Conran
Tags: No tags

Comments are closed.