Kubernetes Security Architecture Explained

Kubernetes Security Architecture

Network Insight / Engineering Guide 12 / Kubernetes Security

Kubernetes Security Architecture: Protecting Cloud-Native Workloads with Network Policies, Cilium and Zero Trust

Kubernetes provides a powerful platform for deploying, scaling and operating containerised applications. However, the same distributed architecture that makes Kubernetes flexible also introduces a complex security environment involving APIs, worker nodes, pods, network connections, workload identities and software supply chains.

Securing Kubernetes requires more than protecting the cluster perimeter. Modern production environments must control how workloads are deployed, which identities can access cluster resources, how services communicate, and what happens when a container is compromised.

This engineering guide explores how Kubernetes security controls operate together, from API authentication and admission control to network microsegmentation, Cilium eBPF enforcement, workload identity, runtime protection and continuous security observability.

KUBERNETES SECURITY ARCHITECTURE Identity · Policy · Network Enforcement · Runtime Protection USERS / CI-CD Authentication / Authorisation API SERVER RBAC / Admission / Audit CONTROL PLANE OBSERVABILITY Audit / Hubble / Runtime KUBERNETES CLUSTER — WORKLOAD SECURITY WORKER NODE A WORKER NODE B FRONTEND API SERVICE DATA Pod identity Policy enforced Service identity Restricted access CILIUM / eBPF ENFORCEMENT CILIUM / eBPF ENFORCEMENT DEFENCE IN DEPTH · LEAST PRIVILEGE · ZERO TRUST

Why Kubernetes Security Requires a Layered Architecture

Kubernetes is not a single security boundary. It is a collection of distributed components, each with different responsibilities and potential attack surfaces.

The Kubernetes API server controls access to cluster resources. Admission mechanisms evaluate requests before supported objects are persisted. Worker nodes execute container workloads. Networking components determine how pods and services communicate. Runtime controls restrict what processes can do after deployment.

A secure architecture must coordinate these layers rather than relying on any single security mechanism.

01
Control Plane Security

Protect API access, authentication, authorisation, admission policies and audit events.

02
Network Security

Enforce workload communication boundaries through network policies and microsegmentation.

03
Workload Identity

Authenticate workloads and authorise service-to-service access using appropriate identities.

04
Runtime Protection

Reduce container privileges and monitor suspicious runtime behaviour.

05
Supply Chain Security

Validate container images, dependencies, deployment artifacts and software integrity.

06
Security Observability

Correlate network flows, audit logs, runtime events and security detections.

Important engineering distinction: Kubernetes NetworkPolicy resources describe network traffic rules, but enforcement depends on the installed networking implementation. Cilium provides an eBPF-based datapath and additional policy capabilities, while workload authentication and application-layer authorisation require appropriate identity and security controls. Network policies alone do not establish a complete Zero Trust architecture.

Security Is an End-to-End Engineering Responsibility

Production Kubernetes environments frequently combine managed cloud infrastructure, Kubernetes control-plane services, container runtimes, third-party networking components, application frameworks and CI/CD pipelines.

Security ownership therefore varies depending on the deployment model. In a managed Kubernetes service, the cloud provider may operate parts of the control plane, while the organisation remains responsible for workload configuration, identities, network policies, application security and other customer-managed controls.

Infrastructure Protection

Protect cluster access, node configuration, network boundaries, secrets and software deployment pipelines.

Workload Protection

Apply least privilege, restrict service communication, validate deployment artifacts and monitor runtime activity.

Zero Trust principle: A workload should not receive unrestricted access merely because it runs inside the same Kubernetes cluster or namespace. Access decisions should be based on verified identity, explicit authorisation, least privilege and relevant security context.

What This Engineering Guide Will Teach

The following nine sections explain how to design, enforce, observe and troubleshoot Kubernetes security in production. We will examine NetworkPolicy, Cilium, eBPF, Zero Trust, RBAC, admission control, runtime security, secrets, software supply chains and incident response, with an emphasis on real traffic behaviour, engineering trade-offs and operational diagnostics.

Network Insight · Engineering Guide 12

Kubernetes Security Architecture

Explore nine interconnected security layers, from workload isolation and network enforcement to Zero Trust identity, secrets protection, runtime detection and production incident response. Select a component to explore its engineering section.

⎈ KUBERNETES SECURITY CONTROL PLANE
Aligned icon cards · Select a security control
Nine Engineering Sections
SELECTED SECTION 01 / 09

Security Architecture Fundamentals

Understand Kubernetes trust boundaries, control-plane security, worker-node protection and defence in depth.

Network Insight / Kubernetes Security / Section 01

Kubernetes Security Architecture Fundamentals: Control Plane, Worker Nodes, Pods and Trust Boundaries

Kubernetes security begins with understanding the architecture of the cluster itself. A Kubernetes environment contains several distinct security domains, including the control plane, worker nodes, container workloads, networking components, storage systems and external management interfaces.

Each domain introduces different attack surfaces and enforcement mechanisms. A secure deployment must protect the interfaces between these components while limiting the privileges available to users, services and applications.

1.1 — Kubernetes Security Architecture

The Kubernetes control plane manages cluster state and workload orchestration. Worker nodes execute containers and provide the networking and runtime environment required by applications.

KUBERNETES SECURITY TRUST BOUNDARIES USER / CI-CD API requests API SERVER Authentication / RBAC / Admission ETCD Cluster state WORKLOAD EXECUTION DOMAIN WORKER NODE A WORKER NODE B POD A POD B POD C POD D ✓ AUTHENTICATION · AUTHORISATION · ISOLATION · POLICY

Security interpretation: The API server is the primary control-plane entry point for Kubernetes API operations. Authentication, authorisation and admission control govern requests, while worker-node security and networking controls protect workload execution and communication. These are separate enforcement boundaries that must be secured together.

1.2 — Kubernetes API Request Security

A Kubernetes API request passes through multiple security stages before the requested operation is completed.

Kubernetes API Request Client / Service Account | v TLS Connection | v Authentication | v Authorisation (e.g. RBAC) | v Admission Control | v API Validation / Persistence | v Cluster State / Controllers

Authentication establishes the identity associated with the request. Authorisation determines whether that identity is permitted to perform the requested operation. Admission control can validate or modify supported API requests according to configured policies.

Important: Admission control is not a replacement for authentication or authorisation. It provides an additional policy enforcement stage for applicable API operations. Some operations and API paths have different admission behaviour.

1.3 — Control Plane Security Components

Component Security responsibility
API server Secures Kubernetes API access and request processing
Authentication Establishes user or workload identity
RBAC Controls permitted Kubernetes API actions
Admission control Validates or modifies supported resource requests
etcd Stores cluster state and requires strong access protection
Controller manager Reconciles cluster resources according to desired state
Scheduler Assigns eligible workloads to worker nodes

1.4 — Worker Node and Container Security

Worker nodes provide the execution environment for Kubernetes workloads. They typically run the kubelet, container runtime and networking components required to operate pods.

Compromise of a worker node may expose multiple workloads running on that node. Security controls should therefore protect the host operating system, container runtime, kubelet API, credentials and privileged processes.

Node Hardening

Minimise services, patch the operating system and restrict administrative access.

Kubelet Security

Protect kubelet authentication, authorisation and network exposure.

Container Isolation

Use appropriate security contexts, capabilities, seccomp and runtime controls.

1.5 — Pod Security Boundaries

Pods are Kubernetes deployment and scheduling units, but they should not be treated as strong security boundaries equivalent to separate virtual machines.

Containers within a pod share a network namespace and may share storage volumes and other resources. The level of isolation between pods depends on the operating system, container runtime, security configuration and workload privileges.

Engineering implication: A compromised container may attempt to access shared pod resources, other workloads, node services or Kubernetes credentials. Pod security settings and network segmentation reduce the available attack surface but do not eliminate every possible escape path.

1.6 — Namespaces and Multi-Tenancy

Namespaces organise Kubernetes resources and provide scopes for many API objects. They can support administrative separation, resource policies and network policy selection.

However, namespaces alone do not provide complete security isolation. Multi-tenant environments require additional controls such as RBAC, network policies, admission policies, resource quotas, workload identity and node isolation where appropriate.

Control Purpose
Namespace Logical resource organisation and policy scope
RBAC API permissions for users and service accounts
NetworkPolicy Restricts selected ingress and egress traffic
Pod Security Admission Enforces configured pod security standards
ResourceQuota Limits resource consumption within a namespace
Node isolation Separates sensitive workloads at the infrastructure level

1.7 — Kubernetes Shared Responsibility

Security responsibilities differ between self-managed Kubernetes clusters and managed services such as AKS, EKS and GKE.

Platform Provider

Depending on the service model, the provider may manage control-plane infrastructure, service availability, platform updates and selected underlying infrastructure controls.

Cluster Operator

The organisation remains responsible for its workload security, identities, network configuration, application permissions, secrets and other customer-managed controls.

The exact division of responsibility must be verified against the chosen Kubernetes service and deployment model.

1.8 — Defence-in-Depth Architecture

Defence in depth combines multiple independent security controls so that failure of one mechanism does not automatically expose the entire cluster.

Kubernetes Defence-in-Depth Layer 1 — Identity Authentication / Workload Identity Layer 2 — API Authorisation RBAC / Least Privilege Layer 3 — Admission Security Pod Security / Policy Validation Layer 4 — Network Segmentation NetworkPolicy / Cilium Layer 5 — Runtime Protection Seccomp / Capabilities / Monitoring Layer 6 — Data Protection Secrets / Encryption / KMS Layer 7 — Observability Audit Logs / Network Flows / Alerts

Zero Trust alignment: Defence in depth supports Zero Trust by reducing implicit trust and enforcing explicit access decisions. However, a complete Zero Trust architecture also requires verified identities, appropriate authorisation and continuous evaluation of security context.

1.9 — Security Architecture Validation Checklist

Kubernetes Security Baseline [ ] API server access restricted [ ] Authentication configured securely [ ] RBAC permissions reviewed [ ] Admission policies enforced [ ] etcd access protected [ ] Secrets encryption evaluated [ ] Worker nodes hardened [ ] Kubelet endpoints protected [ ] Pod security controls configured [ ] Network policies deployed [ ] Workload identities restricted [ ] Audit logging enabled [ ] Runtime monitoring operational [ ] Cluster upgrade process documented

The checklist provides a starting point for engineering validation. Production security requirements should also consider regulatory obligations, threat models, deployment architecture and organisation-specific risk.

Section 1 — Engineering Takeaway

Kubernetes security is a layered architecture spanning the control plane, API request pipeline, worker nodes, container runtime, workloads, network communication and identity systems. Understanding these trust boundaries is essential before implementing network microsegmentation, Cilium enforcement or Zero Trust policies.

Network Insight · Engineering Lab 01 · Kubernetes Security

Kubernetes Security Architecture Explorer

Explore Kubernetes trust boundaries, control-plane security, worker-node protection, workload isolation, network enforcement and defence in depth. Select security layers, inspect architecture components and investigate how architectural controls affect common Kubernetes attack scenarios.

Cluster security architecture

0% Enabled controls
0 Protected layers
— Scenario exposure
Ready Investigation state

Defence-in-depth controls

Architecture inspector

Kubernetes Trust Boundaries

Kubernetes security depends on multiple independent controls protecting the control plane, worker nodes, workloads and network paths.

Select a component in the diagram to inspect its security responsibility.
Security investigation console

Architecture findings and engineering recommendations

Engineering interpretation: This lab models architectural security coverage rather than executing Kubernetes API requests or enforcing real policies. Controls such as RBAC, admission, NetworkPolicy and runtime security operate at different stages and cannot substitute for one another. NetworkPolicy enforcement requires a compatible network implementation. A compromised workload identity may still possess legitimately granted permissions. Security outcomes depend on policy details, workload configuration, implementation and the attacker's capabilities.
Network Insight / Kubernetes Security / Section 02

Kubernetes Network Policies and Microsegmentation: Ingress, Egress, Default-Deny and Traffic Enforcement

Kubernetes networking is designed to allow workloads to communicate across a cluster. In a conventional deployment without enforced network isolation, pods can often reach other pods regardless of their namespace or the worker node on which they run.

This connectivity simplifies application deployment, but it also creates an opportunity for lateral movement if a workload becomes compromised. An attacker who gains access to one container may attempt to reach databases, internal APIs, administrative endpoints or other services.

Kubernetes NetworkPolicy provides a declarative mechanism for controlling selected ingress and egress traffic. When supported and enforced by the cluster networking implementation, policies can restrict communication to explicitly approved sources, destinations and ports.

2.1 — Kubernetes Microsegmentation Architecture

Microsegmentation divides the cluster into smaller communication domains. Instead of assuming that all internal workloads are trusted, engineers define which application components require connectivity and which communication paths should remain blocked.

KUBERNETES NETWORK POLICY ENFORCEMENT Namespace isolation · Pod selectors · Ingress / Egress FRONTEND APPLICATION DATABASE Namespace: web Namespace: app Namespace: data WEB POD API POD DB POD app=frontend app=api app=database × DIRECT WEB → DATABASE DENIED ALLOW REQUIRED APPLICATION FLOWS · DENY UNNECESSARY ACCESS

Architecture interpretation: The diagram represents an intended policy outcome: frontend workloads can reach the API, API workloads can reach the database, and direct frontend-to-database communication is blocked. In a real deployment, both ingress and egress isolation must be evaluated, and the installed CNI must support NetworkPolicy enforcement. Namespace boundaries alone do not create these restrictions.

2.2 — How Kubernetes NetworkPolicy Works

A NetworkPolicy is a namespaced Kubernetes resource. Its podSelector identifies the pods to which the policy applies. The policyTypes field identifies whether the policy controls ingress, egress or both.

For an isolated direction, traffic is allowed only when it matches an applicable policy rule, subject to the behaviour of the networking implementation and traffic outside the NetworkPolicy model.

NetworkPolicy evaluation Source Pod | v Source Egress Policy | | Allowed? v Cluster Network / CNI | v Destination Ingress Policy | | Allowed? v Destination Pod For ordinary pod-to-pod traffic: Required source egress permission AND Required destination ingress permission

If a source pod is not isolated for egress, its egress traffic is permitted by NetworkPolicy unless another control blocks it. Likewise, a destination pod that is not isolated for ingress accepts traffic unless another control prevents it.

Important: Kubernetes NetworkPolicies are additive. Standard NetworkPolicy does not provide an explicit deny rule that overrides an allow. Multiple applicable policies contribute to the set of permitted traffic.

2.3 — Default-Deny Network Policies

A default-deny policy establishes a restrictive starting point by selecting pods and isolating their traffic. Engineers then add narrowly scoped allow policies for required communication.

Default-Deny Ingress and Egress
apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-all namespace: app spec: podSelector: {} policyTypes: - Ingress - Egress

This policy selects all pods in the app namespace and isolates both ingress and egress. Because no allow rules are specified, it permits no ordinary ingress or egress traffic under standard NetworkPolicy semantics.

However, some traffic may remain outside NetworkPolicy enforcement, including certain node-related communication. Behaviour must be validated against the CNI and platform.

Production consideration: Applying default-deny egress without allowing DNS, required APIs, monitoring or external dependencies can break applications. Build and validate the required communication inventory before enforcing restrictive policies.

2.4 — Pod Selectors and Namespace Selectors

NetworkPolicy rules use label selectors to identify which peers are permitted. These selectors can match pods within a namespace or combine namespace and pod labels to select workloads across namespaces.

Selector Purpose Engineering example
podSelector Select pods by label Allow app=frontend
namespaceSelector Select namespaces by label Allow environment=production
Combined selectors Select specific pods within selected namespaces Allow app=api in team=payments namespaces
ipBlock Select an IP CIDR range Allow approved external network ranges
ports Limit allowed transport ports and protocols Allow TCP 443

Selector distinction: A single peer entry containing both namespaceSelector and podSelector matches pods satisfying both conditions. Separate peer entries are alternatives. Confusing these structures can unintentionally broaden access.

2.5 — Allow Frontend Traffic to the API

The following policy permits TCP port 8080 ingress to API pods in the app namespace from frontend pods in namespaces labelled role=web.

apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: allow-frontend-to-api namespace: app spec: podSelector: matchLabels: app: api policyTypes: - Ingress ingress: - from: - namespaceSelector: matchLabels: role: web podSelector: matchLabels: app: frontend ports: - protocol: TCP port: 8080

This policy permits the specified ingress traffic. It does not automatically grant the frontend pod egress permission if that pod is already isolated for egress. A corresponding egress allow rule may therefore be required.

2.6 — DNS Egress and Essential Dependencies

DNS is a common source of application failures after default-deny egress is enabled. Workloads may be unable to resolve Kubernetes Service names or external dependencies even though their application code and service configuration are otherwise correct.

A production policy should identify the actual DNS service, namespace labels, pod labels and transport requirements rather than assuming a universal CoreDNS deployment configuration.

Typical DNS investigation 1. Identify the cluster DNS service 2. Inspect its namespace and pod labels 3. Verify DNS destination IP and port 4. Permit required UDP/TCP port 53 5. Validate queries from isolated pods 6. Review CNI flow logs for denied traffic

Engineering note: Standard Kubernetes NetworkPolicy does not provide DNS-name-based egress filtering. Some CNIs, including Cilium, offer additional FQDN-aware policy capabilities through their own policy resources.

2.7 — NetworkPolicy Enforcement and the CNI

Kubernetes stores NetworkPolicy objects through its API, but the API server does not itself filter application packets. Actual traffic enforcement is performed by a compatible networking implementation.

Kubernetes API

Stores the declarative policy configuration, including selectors, directions, peers and ports.

Network Enforcement

The installed CNI or associated datapath translates supported policy rules into packet-forwarding decisions.

Implementations may use Linux networking facilities, eBPF or other platform-specific mechanisms. Capabilities and treatment of special traffic cases differ between implementations.

2.8 — Production Microsegmentation Design

Effective microsegmentation begins with application communication discovery. Engineers must identify legitimate service dependencies before constructing policies.

01
Discover Traffic

Map application flows, service dependencies and required external endpoints.

02
Classify Workloads

Use consistent labels for applications, environments and security domains.

03
Define Trust Boundaries

Separate frontend, application, data and management traffic.

04
Apply Default Deny

Introduce isolation with staged testing and operational safeguards.

05
Allow Required Flows

Grant only the sources, destinations and ports needed.

06
Validate Enforcement

Test permitted and denied paths and monitor application health.

2.9 — NetworkPolicy Troubleshooting

Symptom Potential cause Investigation
Pod cannot reach API Missing egress or ingress allow rule Check policies selecting both endpoints
DNS resolution fails DNS traffic blocked by egress isolation Inspect DNS flow and allow rules
Unexpected access remains possible Another additive allow policy or unsupported traffic case Review all applicable policies and CNI behaviour
Namespace selector does not match Incorrect namespace labels Inspect namespace metadata
Policy exists but is ineffective CNI lacks required enforcement capability Verify network plugin support and health
Service traffic behaves unexpectedly Address translation or implementation-specific processing Trace service forwarding and policy enforcement points
Kubernetes NetworkPolicy Diagnostics kubectl get networkpolicy -A kubectl describe networkpolicy \ -n app allow-frontend-to-api kubectl get pods -n app --show-labels kubectl get namespaces --show-labels kubectl get pods -n app -o wide kubectl get services -A kubectl get endpointslices -A

When Cilium is installed, Hubble flow visibility can provide additional information about forwarded and dropped traffic. Exact commands depend on the installed version and tooling.

2.10 — Engineering Security Checklist

Kubernetes Microsegmentation Checklist [ ] CNI NetworkPolicy support verified [ ] Namespace labels reviewed [ ] Pod labels standardised [ ] Application dependencies documented [ ] Default-deny strategy defined [ ] Ingress rules restricted [ ] Egress rules restricted [ ] DNS access validated [ ] Management traffic considered [ ] Cross-namespace traffic tested [ ] Allowed flows verified [ ] Denied flows verified [ ] Network flow telemetry enabled [ ] Policy changes version-controlled [ ] Recovery and rollback tested
Section 2 — Engineering Takeaway

Kubernetes NetworkPolicy provides a foundational mechanism for network microsegmentation, ingress and egress isolation, least-privilege connectivity and reduced lateral-movement opportunities. Effective deployment requires accurate workload labels, a compatible enforcement implementation, complete application dependency mapping and continuous validation of both permitted and denied traffic.

Network Insight · Engineering Lab 02 · Kubernetes Security

Network Policy Traffic Enforcement Simulator

Build a Kubernetes microsegmentation policy environment. Select source and destination pods, test TCP or UDP traffic, inspect namespace and label selectors, and observe how ingress and egress isolation combine to permit or deny a connection.

Traffic evaluation workspace

0 Enabled policies
— Ingress decision
— Egress decision
— Packet result
Select a traffic flow and evaluate its NetworkPolicy decision.

NetworkPolicy configuration

Enable or disable the policy objects below. Policies are additive: multiple policies can select the same pod without rule-order precedence.

Policy inspector

NetworkPolicy Evaluation

NetworkPolicy rules select destination or source pods using labels and namespaces.

A connection must satisfy the applicable source egress and destination ingress requirements.
Policy decision console

Engineering findings

Policy semantics: A pod is isolated for ingress or egress if at least one enabled NetworkPolicy selects that pod for the corresponding policy type. If isolated, traffic is allowed only when at least one selecting policy permits it in that direction. Policies are additive and do not have ordered deny rules. The simulator models direct pod-to-pod traffic using fixed labels and namespaces; Service NAT, hostNetwork traffic, node-specific behaviour, CNI implementation details and connection tracking are outside this model. NetworkPolicy enforcement requires a compatible network plugin.
Network Insight / Kubernetes Security / Section 03

Cilium and eBPF Security Architecture: Kernel Enforcement, Workload Identity, L3–L7 Policy and Hubble Observability

Cilium is a cloud-native networking, security and observability platform that uses extended Berkeley Packet Filter (eBPF) technology to implement networking functions and enforce security policies within supported Linux environments.

Rather than relying exclusively on traditional packet-filtering chains, Cilium can attach eBPF programs to supported kernel networking hooks, allowing packet processing and policy decisions to occur at specific points in the Linux networking datapath.

A major architectural feature is Cilium's use of security identities derived from workload labels. This allows policies to express relationships between workloads without depending solely on their changing IP addresses.

3.1 — Cilium eBPF Security Datapath Architecture

Cilium separates policy configuration from packet enforcement. Kubernetes resources and Cilium policy objects express desired security behaviour. The Cilium agent processes this configuration and programs the datapath accordingly.

CILIUM / eBPF SECURITY DATAPATH Declarative policy → Identity mapping → Kernel enforcement POLICY API CILIUM AGENT HUBBLE NetworkPolicy / CNP Identity / Policy maps Flow observability LINUX KERNEL / NETWORK DATAPATH SOURCE POD Identity: frontend DESTINATION POD Identity: api eBPF ENFORCEMENT Endpoint identity lookup Policy map evaluation POLICY ALLOW IDENTITY-AWARE NETWORK SECURITY Logical architecture — actual eBPF hooks and paths depend on deployment mode

Architecture interpretation: The diagram shows a logical enforcement flow. Actual Cilium packet processing depends on the endpoint type, kernel capabilities, routing configuration, selected datapath features and policy type. eBPF programs may execute at different kernel hooks, and some L7 policies involve proxy processing.

3.2 — What eBPF Changes in Kubernetes Networking

eBPF enables verified programs to run at supported points within the Linux kernel. These programs can inspect packet metadata, perform lookups in kernel maps and apply networking decisions.

Cilium uses eBPF to implement supported networking, load-balancing, security and observability functions.

01
Kernel Enforcement

Policy decisions can occur in supported kernel datapath hooks.

02
Identity-Based Security

Workloads can be represented by label-derived security identities.

03
Policy Maps

Kernel maps store data used by eBPF programs for enforcement.

04
Flow Visibility

Network events can be exposed through Cilium and Hubble.

05
Service Processing

eBPF can support Kubernetes service load-balancing functions.

06
Extended Policy

Cilium adds capabilities beyond standard Kubernetes NetworkPolicy.

3.3 — Cilium Security Identities

Cilium assigns security identities to endpoints based on selected security-relevant labels. Endpoints sharing an equivalent security label set may share the same identity.

This approach allows policies to express relationships between workload categories rather than relying exclusively on individual pod IP addresses.

Illustrative workload identities Frontend Pod namespace: web app: frontend identity: 12001 API Pod namespace: app app: api identity: 12002 Database Pod namespace: data app: database identity: 12003 Policy intention: Frontend → API: ALLOW Frontend → Database: DENY API → Database: ALLOW

Important: The identity numbers above are illustrative. Cilium identity allocation and label handling depend on the environment and configuration. An identity is not equivalent to a Kubernetes ServiceAccount identity or a cryptographic workload identity.

3.4 — Cilium Policy Enforcement and eBPF Maps

Cilium agents translate supported policies into datapath state. eBPF programs consult kernel maps containing relevant endpoint, identity, connection and policy information.

Logical packet policy decision Packet arrives | v Identify endpoint and direction | v Determine source / destination identity | v Consult applicable policy state | +---- Allowed ----> Continue forwarding | +---- Denied -----> Drop / report event

The exact lookup sequence varies by traffic direction, endpoint type and enabled features. Not every packet necessarily follows an identical chain of operations.

3.5 — Layer 3, Layer 4 and Layer 7 Security

Layer Policy capability Example
Layer 3 Workload identity, endpoint or network-based controls Frontend can reach API workloads
Layer 4 Protocol and transport port restrictions Allow TCP port 8080
Layer 7 Supported application protocol rules Allow selected HTTP methods and paths

Layer 7 policy requires protocol-aware inspection. Depending on the policy and traffic type, Cilium can integrate with Envoy proxy processing to enforce application-layer rules.

Encryption limitation: Application-layer inspection of encrypted traffic requires an appropriate mechanism for accessing the relevant application data. Cilium cannot inspect arbitrary encrypted HTTP payloads simply because an eBPF program observes the packets.

3.6 — CiliumNetworkPolicy and Extended Security Controls

Cilium supports standard Kubernetes NetworkPolicy resources and its own policy APIs, including CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy.

These additional resources provide Cilium-specific capabilities such as supported L7 rules, identity-aware selectors and selected DNS-aware egress controls.

Illustrative CiliumNetworkPolicy apiVersion: cilium.io/v2 kind: CiliumNetworkPolicy metadata: name: api-http-policy namespace: app spec: endpointSelector: matchLabels: app: api ingress: - fromEndpoints: - matchLabels: app: frontend toPorts: - ports: - port: "8080" protocol: TCP rules: http: - method: GET path: "/health"

This example illustrates an HTTP policy structure. Cross-namespace selection, label scoping and deployment behaviour must be verified against the installed Cilium version and policy documentation.

3.7 — Hubble Network Observability

Hubble provides network flow visibility for Cilium-managed environments. It can expose information about endpoints, identities, traffic directions, policy verdicts and supported protocol details.

Flow Records

Observe network traffic between workloads and endpoints.

Policy Verdicts

Investigate forwarded and dropped communication.

Service Visibility

Relate observed traffic to Kubernetes workloads and services.

Observability distinction: Hubble visibility depends on the configured deployment, supported protocols, available flow events and telemetry settings. It should not be treated as a complete packet-capture replacement for every troubleshooting case.

3.8 — Production Cilium Troubleshooting

Symptom Potential cause Investigation
Traffic unexpectedly denied Policy selector or identity mismatch Inspect endpoint identity and policy verdict
Policy not enforced Configuration or datapath issue Verify agent health and policy state
Service connectivity failure Service translation, routing or policy issue Trace service and backend traffic
Unexpected L7 rejection HTTP rule or proxy processing issue Inspect L7 policy and proxy events
Missing flow visibility Hubble configuration or event limitations Verify Hubble status and collection settings
Node-to-node communication failure Routing, tunnelling, MTU or firewall issue Inspect datapath and underlay connectivity
Illustrative Cilium diagnostics cilium status cilium connectivity test kubectl get ciliumnetworkpolicies -A kubectl get ciliumclusterwidenetworkpolicies hubble observe hubble observe --verdict DROPPED hubble status

These commands assume the corresponding CLI tools are installed and configured. Available flags and behaviour vary by Cilium and Hubble version.

3.9 — Cilium Security Engineering Checklist

Cilium / eBPF Security Checklist [ ] Cilium agent health verified [ ] Endpoint identities understood [ ] Security labels standardised [ ] NetworkPolicy enforcement tested [ ] CiliumNetworkPolicy rules reviewed [ ] L3 / L4 controls validated [ ] L7 proxy requirements understood [ ] Hubble observability enabled [ ] Dropped flows investigated [ ] Service forwarding validated [ ] MTU and routing configuration checked [ ] Kernel compatibility verified [ ] Policy changes version-controlled [ ] Upgrade compatibility tested
Section 3 — Engineering Takeaway

Cilium extends Kubernetes network security through eBPF-based datapath enforcement, label-derived security identities, L3–L7 policy capabilities and Hubble observability. Its value comes from connecting declarative workload policy with efficient kernel-level network processing and actionable traffic visibility. Successful production deployment requires careful identity design, policy validation and datapath troubleshooting.

Network Insight · Engineering Lab 03 · Kubernetes Security

Cilium eBPF Packet Inspection Visualiser

Follow a packet through a simplified Cilium datapath. Inspect endpoint security identities, eBPF policy lookups, optional L7 proxy enforcement, forwarding decisions and Hubble-style flow observations.

Packet inspection configuration

1101 Source identity
1201 Destination identity
0/0 Datapath stage
Ready Packet verdict
Datapath ready

Inspect a packet to construct the datapath stages.

eBPF policy map inspector

Datapath component inspector

Cilium Security Architecture

Cilium associates endpoint security identities with workload labels and evaluates policy using its datapath and supporting components.

Select a component in the diagram to inspect its role.
Hubble-style flow console

Packet inspection findings

Datapath model: This visualisation is a conceptual teaching model, not a kernel-level emulator. Security identities are simplified fixed examples; actual Cilium identities are derived from relevant endpoint labels. L3/L4 policy checks are represented as a logical policy lookup. L7 HTTP enforcement requires a supported proxy-redirection configuration, commonly using Envoy, and does not mean arbitrary encrypted traffic can be inspected without the appropriate termination or visibility. Hubble observations are synthetic and are not collected from a live cluster.
Network Insight / Kubernetes Security / Section 04

Zero Trust Security for Kubernetes: Workload Identity, mTLS, Least Privilege and Service-to-Service Authorisation

Zero Trust is a security architecture based on the principle that access should not be granted solely because a user, workload or network connection originates from a trusted location. Every access request should be evaluated against explicit identity and authorisation requirements.

In Kubernetes, workloads are dynamic. Pods are created, terminated and rescheduled across worker nodes. Their IP addresses may change frequently, and services communicate across namespaces and cluster boundaries.

A production Zero Trust architecture therefore requires controls that can identify workloads, authenticate communication peers, enforce least-privilege access and provide visibility into service-to-service communication.

4.1 — Kubernetes Zero Trust Architecture

A common Zero Trust design combines workload identities, encrypted communication, service authorisation, network segmentation and security observability.

ZERO TRUST WORKLOAD COMMUNICATION Verify identity · Authenticate peers · Authorise requests WORKLOAD IDENTITY AUTHORITY SPIFFE / SPIRE / Platform Identity IDENTITY ISSUANCE SOURCE WORKLOAD DESTINATION SERVICE FRONTEND POD API POD Authenticated identity Authorised endpoint mTLS Peer authentication Encrypted transport AUTHORISATION POLICY ✓ IDENTITY + ENCRYPTION + AUTHORISATION + OBSERVABILITY

Architecture interpretation: The diagram illustrates a logical Zero Trust service communication model. Identity issuance, certificate distribution, mTLS termination and authorisation enforcement depend on the selected implementation. SPIFFE/SPIRE, service meshes and cloud-native workload identity systems provide different capabilities and integration models.

4.2 — Zero Trust Principles in Kubernetes

01
Verify Identity

Authenticate workloads using appropriate identities and trusted credentials.

02
Least Privilege

Permit only the API actions and service communication required by each workload.

03
Explicit Authorisation

Evaluate requests against defined security policies.

04
Encrypted Communication

Protect sensitive service traffic using appropriate transport security.

05
Segmentation

Restrict unnecessary east-west and north-south communication.

06
Continuous Visibility

Monitor identity, network flows and authorisation decisions.

4.3 — Kubernetes Service Accounts and Workload Identity

Kubernetes ServiceAccounts provide identities for processes running in pods when interacting with the Kubernetes API and compatible integrated systems.

Modern Kubernetes deployments can use projected, bounded service account tokens that have configurable audiences and expiration characteristics.

These tokens can support workload identity federation with external systems when the receiving platform is configured to trust the appropriate issuer and validate the token.

Kubernetes workload identity Pod | v ServiceAccount | v Projected Token | v Trusted Identity Integration | v External Service Authentication

Security distinction: A Kubernetes ServiceAccount token is not automatically an mTLS certificate. Service account authentication, federated cloud identity and SPIFFE-based X.509 identities are related but distinct mechanisms.

4.4 — SPIFFE and SPIRE Architecture

SPIFFE defines a framework for workload identity across heterogeneous systems. Workloads can receive SPIFFE identities represented by SPIFFE IDs.

SPIRE is an implementation of SPIFFE that provides workload attestation and identity issuance capabilities.

Illustrative SPIFFE identity spiffe://example.org/ns/payments/sa/api Trust domain: example.org Workload identity: namespace: payments service account: api Identity material: X.509-SVID or JWT-SVID

SPIFFE Verifiable Identity Documents (SVIDs) can be used by supported applications and infrastructure to authenticate workloads.

Engineering consideration: The SPIFFE ID above is illustrative. Actual identity naming and attestation policies depend on the deployment. Secure identity issuance requires trusted workload and node attestation, protected signing infrastructure and appropriate credential rotation.

4.5 — Mutual TLS for Service-to-Service Communication

Mutual TLS allows both communication peers to authenticate each other using certificates during the TLS handshake.

When correctly configured, mTLS provides encrypted transport, peer authentication and protection against certain network-level impersonation and interception attacks.

Simplified mutual TLS handshake Service A Service B | | |------ ClientHello ------->| |<----- ServerHello --------| |<----- Certificate --------| |------ Certificate ------->| |------ Key Exchange ------>| |<----- TLS Finished -------| |------ TLS Finished ------>| | | |==== Encrypted Traffic ===>|

The exact message sequence depends on the TLS version and authentication configuration. TLS 1.3, for example, uses a different handshake structure from earlier TLS versions.

Important: mTLS authenticates communication peers, but does not by itself decide which application operations those peers may perform. Authorisation policies must still determine whether an authenticated identity can access a particular service or resource.

4.6 — Service Mesh and Zero Trust Enforcement

Service mesh platforms can provide service-to-service security capabilities such as mTLS, workload authentication, traffic policy and telemetry.

Different implementations use different datapath architectures. Some rely on sidecar proxies, while others support alternative proxy placement or ambient models.

Security function Purpose
Workload authentication Verify the identity of service communication peers
mTLS Encrypt traffic and authenticate peers
Authorisation policy Permit or deny selected service requests
Telemetry Observe service communication and policy outcomes
Certificate management Issue, distribute and rotate workload credentials

A service mesh is not mandatory for every Zero Trust deployment. Applications and platform components may implement equivalent controls through other mechanisms.

4.7 — Identity-Aware Authorisation

An authenticated workload should receive only the permissions required to perform its legitimate function.

For example, a frontend service may be permitted to invoke selected API endpoints while being denied access to administrative operations.

Illustrative authorisation model Identity: frontend-service Requested action: GET /api/products Policy: Allow frontend-service to invoke GET /api/products Decision: ALLOW Requested action: DELETE /api/admin/users Policy: No matching permission Decision: DENY

This is a conceptual policy example, not a Kubernetes or service mesh configuration file.

4.8 — Continuous Verification and Credential Rotation

Zero Trust architecture requires security controls to remain effective throughout the workload lifecycle.

Workload credentials should be short-lived where practical, rotated securely and revoked or invalidated when their associated trust conditions no longer apply.

Identity Lifecycle

Issue credentials to validated workloads and manage their expiration.

Policy Lifecycle

Review access rules as applications and dependencies change.

Telemetry Lifecycle

Monitor authentication failures, denied requests and suspicious behaviour.

Operational distinction: Continuous verification does not necessarily mean performing a new cryptographic handshake for every application request. It means maintaining appropriate identity, authorisation and monitoring controls throughout the communication lifecycle.

4.9 — Production Zero Trust Troubleshooting

Symptom Potential cause Investigation
mTLS handshake failure Certificate, trust or protocol mismatch Inspect certificate validity and trust configuration
Workload identity rejected Incorrect issuer, audience or identity mapping Validate token or SVID attributes
Service request denied Authorisation policy mismatch Inspect caller identity and policy decision
Unexpected service access Overly broad authorisation or missing enforcement Review effective policies and traffic paths
Credential rotation failure Identity distribution or renewal issue Inspect identity service and workload logs
Service connectivity failure Network policy, routing or mTLS configuration Separate network reachability from authentication

4.10 — Zero Trust Engineering Checklist

Kubernetes Zero Trust Checklist [ ] Workload identities defined [ ] Service account permissions reviewed [ ] Token audiences restricted [ ] Credential lifetimes controlled [ ] Identity issuance protected [ ] mTLS requirements documented [ ] Certificate rotation tested [ ] Service authorisation enforced [ ] Network segmentation validated [ ] Cross-namespace access reviewed [ ] Authentication failures monitored [ ] Authorisation denials monitored [ ] Identity trust domains documented [ ] Recovery and revocation tested
Section 4 — Engineering Takeaway

Zero Trust Kubernetes security combines verified workload identities, authenticated communication, encrypted transport, explicit authorisation, least-privilege access and continuous security visibility. Network location alone should not determine trust. Effective implementation requires coordinated identity, networking and application-layer security controls.

Network Insight · Engineering Lab 04 · Kubernetes Security

Zero Trust Workload Access Decision Lab

Investigate how Kubernetes workloads establish trust across service boundaries. Evaluate service identity, certificate validity, mTLS, authorisation policy, request scope and continuous verification to determine whether service-to-service access should be permitted.

Workload access investigation

— Identity
— Transport trust
— Authorisation
Ready Access verdict
Zero Trust verification ready

Select a scenario and evaluate its access decision.

Trust boundary inspector

Workload Trust Architecture

Zero Trust requires explicit authentication, authorisation and ongoing validation rather than trusting traffic solely because it originates inside the cluster.

Select an architecture component to inspect its role.

Workload identity

Illustrative identity format
spiffe://cluster.local/ns/web/sa/frontend
Access decision console

Zero Trust engineering findings

Engineering interpretation: This lab models workload-level trust using SPIFFE-style identities, mTLS and application authorisation. Kubernetes does not provide service-mesh mTLS or SPIFFE identities automatically; these capabilities require an appropriate implementation. A Kubernetes service account identifies a workload to the Kubernetes API, while a mesh or workload-identity system may establish separate service-to-service credentials. Certificate validation, revocation, policy propagation and continuous verification depend on the chosen implementation. The model is illustrative and does not issue real certificates or enforce live access policies.
Network Insight / Kubernetes Security / Section 05

Kubernetes RBAC, Service Accounts and API Security: Authentication, Authorisation, Admission and Privilege Escalation

The Kubernetes API server is one of the most security-critical components in a cluster. It provides the primary interface through which users, controllers, automation systems and workloads interact with Kubernetes resources.

A compromised identity with excessive API permissions may be able to read sensitive configuration, modify workloads, create privileged pods or alter cluster security controls.

Production API security therefore requires strong authentication, least-privilege authorisation, carefully controlled service accounts, admission policies, audit logging and continuous permission review.

5.1 — Kubernetes API Security Decision Pipeline

A Kubernetes API request passes through a sequence of security and validation stages. Each stage performs a different function and may reject the request before it reaches the requested operation.

KUBERNETES API SECURITY PIPELINE Identity verification · Permission evaluation · Admission enforcement API CLIENT User / ServiceAccount / Automation AUTHENTICATION AUTHORISATION ADMISSION Verify identity RBAC / Authorisers Validate / Mutate API OPERATION Validation / Persistence / Response REQUEST DENIED Security rejection AUTHENTICATE → AUTHORISE → ADMIT → EXECUTE

Architecture interpretation: The diagram is a simplified representation of API request processing. Kubernetes also performs request parsing, validation and other processing steps. Admission applies to supported operations, and read-only requests do not follow the same admission path as resource creation or modification.

5.2 — Kubernetes Authentication

Authentication determines the identity associated with an API request. Kubernetes supports several authentication mechanisms, depending on cluster configuration.

01
Client Certificates

Authenticate clients using trusted X.509 certificates.

02
Bearer Tokens

Authenticate supported service account or external identity tokens.

03
OIDC Integration

Integrate Kubernetes API authentication with a trusted identity provider.

Authentication establishes who the request represents. It does not automatically grant permission to perform an operation.

5.3 — Kubernetes RBAC Architecture

Role-Based Access Control (RBAC) is a Kubernetes authorisation mechanism that associates permissions with users, groups and service accounts.

RBAC rules describe allowed API operations using verbs, API groups, resources and optional resource names.

RBAC resource Scope Purpose
Role Namespace Defines permissions within a namespace
ClusterRole Cluster-wide definition Defines permissions for cluster-scoped or namespaced resources
RoleBinding Namespace Grants a Role or ClusterRole within one namespace
ClusterRoleBinding Cluster Grants a ClusterRole across applicable cluster scope

Important: A ClusterRole does not automatically grant cluster-wide access. The binding determines where its permissions apply. A RoleBinding can reference a ClusterRole while granting namespaced permissions only within the RoleBinding's namespace.

5.4 — Least-Privilege RBAC Example

Consider an application controller that needs to read pods within the production namespace but does not need to create, modify or delete them.

apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: namespace: production name: pod-reader rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list", "watch"]

This Role defines read-only permissions for pods within the production namespace.

apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: namespace: production name: pod-reader-binding subjects: - kind: ServiceAccount name: monitoring-agent namespace: production roleRef: kind: Role name: pod-reader apiGroup: rbac.authorization.k8s.io

The RoleBinding grants those permissions to the specified service account.

Engineering implication: The service account can perform the permitted pod read operations within the namespace, assuming no other authorisation mechanism rejects the request. The binding does not itself grant permission to modify or delete pods.

5.5 — Service Account Token Security

Service accounts allow workloads to authenticate to the Kubernetes API and compatible integrated systems.

Modern Kubernetes deployments typically use projected, time-bound service account tokens rather than automatically creating long-lived token Secrets.

Service Account Token Lifecycle Pod Created | v ServiceAccount Selected | v Projected Token Issued | v Token Mounted into Pod | v Application Uses Token | v Token Rotation / Expiration

Workloads that do not need Kubernetes API access should avoid unnecessary service account token exposure.

apiVersion: v1 kind: Pod metadata: name: static-worker spec: automountServiceAccountToken: false containers: - name: worker image: example/worker:v1

Security consideration: Disabling automatic token mounting reduces one credential exposure path, but it does not prevent all forms of Kubernetes API access. Other credentials or network access may still exist.

5.6 — Admission Controllers and Policy Enforcement

Admission controllers provide additional control over supported API operations after authentication and authorisation.

They can validate requested resources or modify them according to configured platform policies.

Mutating Admission

May modify supported resource objects before they are persisted.

Validating Admission

May accept or reject supported requests based on configured requirements.

Examples include Pod Security Admission, ValidatingAdmissionPolicy and webhook-based admission controls. Availability and capabilities depend on Kubernetes version and configuration.

5.7 — RBAC Privilege Escalation Risks

Overly broad Kubernetes permissions can allow an attacker or compromised workload to gain access beyond its intended responsibilities.

Permission Potential security impact
Read Secrets May expose credentials or sensitive application data
Create Pods May allow workloads with dangerous configurations if admission controls permit them
Update Deployments May allow modification of application containers or configuration
Create RoleBindings Can expand access within Kubernetes RBAC escalation restrictions
Impersonate May allow requests to be evaluated as another identity
Bind / Escalate Special permissions that can bypass selected RBAC escalation safeguards

RBAC escalation protection: Kubernetes includes restrictions designed to prevent users from granting permissions they do not already possess. Special permissions such as bind and escalate can affect these restrictions. Effective privilege depends on the complete set of authorisation rules and admission controls.

5.8 — Kubernetes API Audit Logging

Kubernetes audit logging records selected API activity according to configured audit policies.

Audit records can help security teams investigate changes to resources, access attempts and administrative operations.

Who

Identify the user or service account associated with a request.

What

Record the requested operation and affected resource.

Outcome

Inspect response status, timestamps and available request metadata.

Operational consideration: Audit log content and volume depend on the configured audit policy and backend. Sensitive data exposure, retention, access control and storage capacity must be considered.

5.9 — Production RBAC and API Troubleshooting

Symptom Potential cause Investigation
HTTP 401 Unauthorized Authentication failure or missing credentials Inspect authentication configuration and token validity
HTTP 403 Forbidden Insufficient authorisation or admission rejection Inspect response message, RBAC and admission policies
Service account cannot list pods Missing RoleBinding or incorrect namespace Review Role and RoleBinding resources
Unexpected cluster-wide access Broad ClusterRoleBinding or additional permissions Review effective RBAC bindings
Pod creation rejected Admission policy or resource validation Inspect API response and admission events
Token authentication fails Expired token, issuer or audience mismatch Validate token properties and authentication configuration
Kubernetes RBAC Diagnostics kubectl auth can-i get pods \ -n production kubectl auth can-i list pods \ -n production \ --as=system:serviceaccount:production:monitoring-agent kubectl get roles -A kubectl get rolebindings -A kubectl get clusterroles kubectl get clusterrolebindings kubectl describe rolebinding \ -n production pod-reader-binding

The impersonation example requires appropriate permission to impersonate the specified service account. A successful can-i check does not replace testing the complete request path, including admission controls.

5.10 — Kubernetes API Security Checklist

Kubernetes API Security Checklist [ ] API endpoint access restricted [ ] Authentication methods reviewed [ ] RBAC least privilege enforced [ ] ClusterRoleBindings audited [ ] Service accounts scoped correctly [ ] Token mounting restricted [ ] Token audiences validated [ ] Admission policies configured [ ] Privileged pod creation controlled [ ] Secret access permissions reviewed [ ] Impersonation permissions reviewed [ ] Bind / Escalate permissions audited [ ] Audit logging configured [ ] Audit retention protected [ ] Access reviews scheduled [ ] Incident response procedures tested
Section 5 — Engineering Takeaway

Kubernetes API security depends on strong authentication, least-privilege RBAC, carefully managed service accounts, admission enforcement and comprehensive audit visibility. The most important engineering objective is to ensure that every identity receives only the permissions necessary for its legitimate operational role.

Network Insight · Engineering Lab 05 · Kubernetes Security

Kubernetes RBAC Permission Investigation Lab

Investigate Kubernetes API authorisation. Select service accounts, namespaces, API verbs and resources; inspect RoleBindings and ClusterRoleBindings; and determine whether a request is permitted by the effective RBAC rules.

API authorisation investigation

— Authentication
0 Matching bindings
— RBAC decision
Ready API request
RBAC evaluation ready

Select an API request and evaluate its permissions.

RBAC binding configuration

Authorisation inspector

Kubernetes API Authorisation

Kubernetes RBAC evaluates requests using the permissions granted through RoleBindings and ClusterRoleBindings.

Select a component to inspect its role.
API decision console

RBAC engineering findings

RBAC semantics: Kubernetes RBAC permissions are additive; RBAC does not implement explicit deny rules. A RoleBinding grants permissions within its namespace, while a ClusterRoleBinding grants permissions across the appropriate scope. A RoleBinding can reference a ClusterRole and still grants its namespaced permissions only within the binding namespace. Authentication precedes authorisation, and admission may impose additional restrictions after an authorised request. Kubernetes also applies privilege-escalation protections to relevant role and binding operations. This educational model simplifies those protections and does not execute real API requests.
Network Insight / Kubernetes Security / Section 06

Pod Security, Container Isolation and Runtime Protection: Kubernetes Admission, Linux Security and Container Escape Prevention

Kubernetes containers execute application processes using operating-system isolation mechanisms. Unlike conventional virtual machines, standard Linux containers running on the same worker node share the host kernel.

This architecture provides efficient resource utilisation and rapid workload deployment, but it also introduces security considerations. A vulnerable application, excessive Linux capabilities, privileged container or exposed host resource may create opportunities for privilege escalation or movement beyond the intended container boundary.

A production container security architecture therefore combines admission-time configuration controls, Linux process isolation, restricted privileges, hardened container runtimes and continuous runtime monitoring.

6.1 — Kubernetes Container Security Architecture

Container security spans several layers. Kubernetes admission policies govern supported workload configurations before execution. The container runtime creates the workload environment, while Linux kernel mechanisms enforce process, resource and access restrictions.

KUBERNETES CONTAINER SECURITY STACK Admission policy · Runtime isolation · Kernel enforcement POD SECURITY ADMISSION Baseline / Restricted / Privileged KUBERNETES WORKER NODE CONTAINER A CONTAINER B Namespaces / Capabilities Seccomp / Security Context Namespaces / Capabilities Seccomp / Security Context SHARED LINUX KERNEL System calls · Access control · Resource management CONFIGURATION CONTROL + KERNEL ISOLATION + RUNTIME MONITORING

Architecture interpretation: The diagram represents standard Linux-container isolation. Sandboxed runtimes may introduce additional isolation mechanisms, including virtualisation boundaries. Admission controls operate before supported pod configurations are accepted, while runtime controls protect executing processes.

6.2 — Kubernetes Pod Security Standards

Kubernetes defines Pod Security Standards that describe different levels of workload security restrictions.

Profile Security posture Typical use
Privileged Unrestricted profile with minimal policy limitations Trusted infrastructure workloads requiring elevated access
Baseline Prevents selected known privilege escalation risks General workloads requiring baseline safeguards
Restricted Applies stronger hardening requirements Security-sensitive application workloads

Pod Security Admission can enforce, audit or warn about violations according to namespace labels and configured policy versions.

apiVersion: v1 kind: Namespace metadata: name: production labels: pod-security.kubernetes.io/enforce: restricted pod-security.kubernetes.io/audit: restricted pod-security.kubernetes.io/warn: restricted

This example configures the namespace to apply the Restricted profile for the selected admission modes. Actual requirements depend on the Kubernetes version and the applicable Pod Security Standards definition.

6.3 — Linux Namespaces and Process Isolation

Linux namespaces provide isolation for selected operating-system resources. Container runtimes use these mechanisms to create separate process environments.

PID Namespace

Controls process identifier visibility and process namespace isolation.

Network Namespace

Provides separate network interfaces, routing tables and related networking state.

Mount Namespace

Controls visibility of filesystem mount points.

IPC Namespace

Separates selected inter-process communication resources.

UTS Namespace

Provides isolation for hostname and domain-name information.

User Namespace

Supports mapping user and group identifiers between namespaces.

Important: Containers in the same Kubernetes pod normally share a network namespace. Other namespace-sharing behaviour depends on pod settings and runtime configuration. Namespace isolation does not remove the shared-kernel security dependency.

6.4 — Linux Capabilities and Privilege Reduction

Linux capabilities divide selected superuser privileges into more granular permissions.

A container should receive only the capabilities required for its application function. Unnecessary capabilities increase the potential impact of a compromised process.

apiVersion: v1 kind: Pod metadata: name: restricted-app spec: containers: - name: app image: example/app:v1 securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL runAsNonRoot: true seccompProfile: type: RuntimeDefault

This configuration requests several common hardening controls. The container image must support running without root privileges and without additional Linux capabilities.

Engineering consideration: Dropping all capabilities and disabling privilege escalation are useful safeguards, but they do not eliminate every container escape or kernel vulnerability. Additional host, runtime and kernel security controls remain necessary.

6.5 — Seccomp System Call Filtering

Seccomp allows Linux processes to operate under restrictions on available system calls.

A seccomp profile can reduce the system call interface available to a container, limiting access to selected kernel functionality.

Container Process | v System Call | v Seccomp Filter | +---- Permitted ----> Kernel | +---- Restricted ---> Action per profile

Kubernetes supports specifying seccomp profiles through the securityContext configuration.

Important: RuntimeDefault behaviour depends on the installed container runtime. A profile may block application operations that require restricted system calls, so compatibility testing is essential.

6.6 — AppArmor and SELinux

Linux Mandatory Access Control mechanisms provide additional restrictions on process behaviour.

AppArmor

Applies profile-based restrictions to supported processes, including controls over filesystem and other operations.

SELinux

Uses security labels and policy rules to enforce Mandatory Access Control decisions.

The available configuration and enforcement behaviour depend on the host operating system, kernel configuration, container runtime and Kubernetes integration.

6.7 — Privileged Containers and Host Access

Privileged containers can receive extensive access to host resources and may bypass important container isolation restrictions.

Other high-risk configurations include unrestricted hostPath mounts, host networking, host process namespaces and excessive Linux capabilities.

Configuration Potential risk
privileged: true Significantly expands container access to host resources
hostPID: true Exposes the host process namespace
hostNetwork: true Places the pod in the host network namespace
hostPath Can expose sensitive host filesystem paths
Additional capabilities May grant powerful kernel-level operations
Running as root Can increase impact when combined with other weaknesses

Some infrastructure components legitimately require elevated access. Such workloads should be isolated, tightly controlled and reviewed individually.

6.8 — Sandboxed Container Runtimes

Sandboxed runtimes introduce additional isolation mechanisms between workloads and the underlying host.

Examples include technologies that use lightweight virtual machines or user-space kernel interception to reduce exposure to the host kernel.

Standard Containers

Share the host kernel and rely on Linux isolation mechanisms.

MicroVM-Based Isolation

Introduces a virtualisation boundary between workloads and the host.

User-Space Isolation

Uses additional runtime mechanisms to mediate selected application operations.

Sandboxing can improve isolation for selected threat models, but it may introduce compatibility, performance and operational trade-offs.

6.9 — Runtime Threat Detection

Admission controls evaluate workload configuration before execution. Runtime detection examines behaviour after containers begin running.

Runtime monitoring may observe process execution, file access, network activity, privilege changes and other supported operating-system events.

Runtime Security Investigation Container Process | v Process / System Call Event | v Runtime Monitoring | v Detection Rule | +---- Expected Behaviour | +---- Suspicious Behaviour | v Security Alert | v Investigation

Runtime detection technologies may use eBPF, kernel audit facilities or other supported telemetry mechanisms.

Detection limitation: Runtime monitoring is not automatically prevention. Whether a detected operation can be blocked depends on the monitoring technology, its enforcement capabilities and the configured response.

6.10 — Production Container Security Troubleshooting

Symptom Potential cause Investigation
Pod rejected at creation Pod Security Admission violation Inspect API rejection and namespace policy labels
Container fails to start Security context or runtime incompatibility Inspect pod events and container logs
Permission denied Filesystem, capability or MAC policy restriction Inspect runtime permissions and host security events
System call rejected Seccomp profile restriction Inspect seccomp configuration and relevant logs
Unexpected host access Privileged configuration or exposed host resources Review pod specification and runtime privileges
Suspicious process execution Potential workload compromise Correlate runtime events, process context and network activity
Kubernetes Runtime Diagnostics kubectl get pods -A kubectl describe pod \ -n production restricted-app kubectl get pod \ -n production restricted-app -o yaml kubectl get events \ -n production --sort-by=.lastTimestamp kubectl logs \ -n production restricted-app kubectl get namespaces --show-labels

Host-level investigation may require additional operating-system tools and runtime-specific diagnostics.

6.11 — Container Security Engineering Checklist

Kubernetes Container Security Checklist [ ] Pod Security Admission configured [ ] Restricted profile evaluated [ ] Privileged workloads inventoried [ ] Host namespaces restricted [ ] HostPath mounts reviewed [ ] Containers run as non-root [ ] Linux capabilities minimised [ ] Privilege escalation restricted [ ] Seccomp profiles configured [ ] AppArmor / SELinux evaluated [ ] Container runtime patched [ ] Worker node kernel maintained [ ] Runtime telemetry enabled [ ] Suspicious process activity monitored [ ] Sandboxed runtimes evaluated [ ] Security changes tested before rollout
Section 6 — Engineering Takeaway

Kubernetes container security requires multiple independent controls: Pod Security Admission, restricted security contexts, Linux namespace isolation, capability reduction, seccomp, Mandatory Access Control, runtime hardening and continuous threat detection. Because standard Linux containers share the host kernel, securing the worker node remains essential to protecting the workloads it runs.

Network Insight · Engineering Lab 06 · Kubernetes Security

Container Runtime Security Explorer

Investigate container isolation, admission security, Linux capabilities, seccomp restrictions, host access and runtime detection. Modify a workload's security posture and observe how different controls affect deployment admission and runtime risk.

Workload security configuration

— Admission
— Isolation posture
— Runtime alert
Ready Investigation
Container security evaluation ready

Select a workload scenario and evaluate its admission and runtime security posture.

Container security controls

Security layer inspector

Container Isolation Architecture

Kubernetes container security combines admission checks, runtime configuration, Linux isolation mechanisms and operational monitoring.

Select an architecture component to inspect its responsibility.
Runtime security console

Container security findings

Security model: Pod Security Admission evaluates Pod Security Standards at admission time. It does not monitor runtime system calls or detect attacks. Runtime detection requires a separate implementation. Linux namespaces isolate resource views, cgroups control resource accounting and limits, seccomp restricts selected system calls, and Linux capabilities divide certain privileged operations. These mechanisms do not provide a complete security boundary against every kernel exploit. This educational model simplifies Kubernetes Pod Security Standards and runtime enforcement; it does not execute containers or perform kernel-level inspection.
Network Insight / Kubernetes Security / Section 07

Kubernetes Secrets, Encryption and Supply Chain Security: KMS, Image Signing, SBOMs and Secure Deployment Pipelines

Kubernetes workloads depend on application images, configuration data, credentials, certificates and external services. Each dependency introduces a potential security risk if its integrity, confidentiality or origin cannot be adequately verified.

An attacker may attempt to compromise a container image, modify a build pipeline, obtain application secrets or deploy an untrusted workload into the cluster.

Production Kubernetes security therefore requires controls across the complete software delivery lifecycle: source integrity, trusted builds, artifact verification, secure registries, admission enforcement, secrets protection and runtime credential management.

7.1 — Secure Kubernetes Software Supply Chain Architecture

A secure software supply chain establishes trust in the components used to build, distribute and execute workloads. The objective is to reduce the likelihood that malicious or unverified software reaches production.

KUBERNETES SOFTWARE SUPPLY CHAIN SECURITY Source integrity · Trusted builds · Artifact verification · Secret protection SOURCE CI BUILD SIGN / SCAN REGISTRY Code integrity Build provenance SBOM / Signature Trusted artifact ADMISSION VERIFICATION Signature / Provenance / Policy KUBERNETES WORKLOAD Verified deployment SECRET MANAGER KMS / Vault / Cloud Secrets VERIFIED ARTIFACTS + PROTECTED CREDENTIALS + CONTROLLED DEPLOYMENT

Architecture interpretation: The diagram shows two related but distinct security paths. Software artifacts move through build, verification and deployment. Credentials should be managed through protected secrets systems and delivered only to authorised workloads. Actual verification and secret-delivery mechanisms depend on the selected platform and tooling.

7.2 — Kubernetes Secrets Architecture

Kubernetes Secrets store small amounts of sensitive data such as passwords, tokens and certificates.

A Secret is a Kubernetes API object. Applications may consume Secret data through environment variables, mounted volumes or other supported integration mechanisms.

Kubernetes Secret Data Flow Secret Created | v Kubernetes API Server | v etcd Storage | v Kubelet Retrieves Secret | v Authorised Pod | v Container Consumption

Secret data should be accessible only to workloads and identities that require it.

Important: Base64 encoding in a Kubernetes Secret manifest is not encryption. Without appropriate encryption at rest and access controls, Secret data may be exposed through API access or underlying storage.

7.3 — Encryption at Rest and etcd Protection

Kubernetes stores cluster state in etcd. This may include Secret objects and other sensitive configuration.

Kubernetes supports encryption at rest for selected API resources when the API server is configured with an encryption provider.

API Access Control

Restricts who can read or modify Secret objects.

Encryption at Rest

Protects selected resource data stored through the API server.

etcd Security

Requires restricted network access, secure transport, backups and operational controls.

Encryption at rest protects stored data but does not replace RBAC, transport encryption or workload-level credential security.

7.4 — Kubernetes KMS Integration

Kubernetes can integrate with an external Key Management Service through a supported KMS provider mechanism.

KMS integration allows encryption operations to depend on externally managed key material, reducing the need to store all encryption secrets directly within the API server configuration.

Illustrative KMS Architecture Kubernetes API Server | v Encryption Provider | v KMS Plugin | v External Key Management Service | v Protected Encryption Key Material

Engineering consideration: KMS availability, latency, key rotation and recovery procedures can affect Kubernetes API operations. The exact architecture depends on the KMS provider version and implementation.

7.5 — External Secrets Management

Organisations often use external secret-management systems to centralise credential storage, rotation and access control.

Examples include cloud secret managers and dedicated secrets platforms.

Secret Synchronisation

A controller retrieves external secrets and creates or updates Kubernetes Secret objects.

Direct Secret Mounting

Supported drivers or integrations expose external secret material to workloads through mounted volumes.

Security distinction: Synchronising an external secret into a Kubernetes Secret means the data also exists in Kubernetes storage. Direct mounting can reduce that persistence path, but application access, node exposure and credential lifecycle still require protection.

7.6 — Container Image Security

Container images contain application code, operating-system packages, libraries and runtime dependencies.

Vulnerable or malicious components can introduce security risks before a workload starts.

01
Minimal Images

Reduce unnecessary packages and attack surface.

02
Vulnerability Scanning

Identify known vulnerabilities in image components.

03
Trusted Registries

Restrict image sources and protect registry access.

04
Immutable References

Use image digests to identify specific artifact content.

05
Image Signing

Verify artifact origin and integrity using trusted signatures.

06
Build Provenance

Record information about how and where artifacts were produced.

7.7 — Software Bill of Materials (SBOM)

A Software Bill of Materials provides an inventory of software components included in an artifact.

SBOMs can support vulnerability management, dependency analysis, incident response and software supply chain transparency.

Illustrative SBOM Inventory Application Image | +-- Application Code | +-- Runtime Libraries | +-- Operating-System Packages | +-- Third-Party Dependencies | +-- Component Versions | +-- Package Identifiers

Important: An SBOM is an inventory, not proof that an image is secure. Its usefulness depends on completeness, accuracy, provenance and integration with vulnerability and risk management processes.

7.8 — Image Signing and Admission Verification

Image signing provides a mechanism for associating an artifact with a cryptographic signature.

Admission policies can require that deployed images satisfy selected verification conditions before workloads are accepted.

Secure Image Admission Deployment Requested | v Image Reference Identified | v Signature / Attestation Verification | +---- Valid and Trusted | | | v | Policy Checks | | | v | Deployment Allowed | +---- Invalid / Untrusted | v Deployment Denied

Verification systems must establish which signing identities, keys or attestation authorities are trusted.

Security limitation: A valid signature proves only the properties supported by the verification scheme and trust policy. It does not guarantee that the signed software contains no vulnerabilities or malicious functionality.

7.9 — Secure CI/CD Pipeline Architecture

The CI/CD pipeline is a high-value target because it can produce artifacts that are later executed in production.

Pipeline security should protect source repositories, build workers, credentials, artifact registries and deployment permissions.

Pipeline stage Security control
Source Protected branches, code review and access controls
Build Isolated build environments and controlled dependencies
Test Security testing and vulnerability analysis
Artifact Digest pinning, signing and provenance records
Registry Access restrictions and artifact retention policies
Deployment Admission verification and least-privilege deployment identities
Runtime Monitoring, vulnerability response and credential rotation

7.10 — Production Secrets and Supply Chain Troubleshooting

Symptom Potential cause Investigation
Pod cannot read Secret Missing Secret, wrong reference or mount issue Inspect pod events and Secret references
External secret not synchronised Controller, identity or provider failure Inspect external secrets controller status
ImagePullBackOff Registry authentication, image reference or network issue Inspect image pull events and registry access
Deployment rejected Signature or admission policy failure Inspect admission response and verification logs
Secret rotation not reflected Application reload or delivery mechanism issue Inspect secret update and workload consumption
Unexpected image vulnerability Outdated dependency or incomplete scanning Review image digest, SBOM and vulnerability data
Kubernetes Supply Chain Diagnostics kubectl get secrets -A kubectl describe pod \ -n production application-pod kubectl get events \ -n production --sort-by=.lastTimestamp kubectl get deployments -A kubectl get pods -A -o wide kubectl get validatingwebhookconfigurations kubectl get mutatingwebhookconfigurations

Avoid displaying or exporting Secret values during routine troubleshooting. Use metadata, events and controlled diagnostics where possible.

7.11 — Secrets and Supply Chain Security Checklist

Kubernetes Supply Chain Checklist [ ] Secret access permissions reviewed [ ] etcd encryption at rest evaluated [ ] KMS integration assessed [ ] Secret rotation procedures defined [ ] External secret access restricted [ ] Image registries controlled [ ] Image digests used where appropriate [ ] Vulnerability scanning integrated [ ] SBOM generation implemented [ ] Build provenance recorded [ ] Image signatures verified [ ] Admission verification configured [ ] CI/CD credentials protected [ ] Deployment identities restricted [ ] Artifact retention policies defined [ ] Supply chain incident response tested
Section 7 — Engineering Takeaway

Kubernetes secrets and supply chain security require coordinated controls across credential storage, encryption, key management, trusted builds, artifact integrity, image registries, admission verification and secure deployment automation. The objective is to ensure that production workloads execute approved software while receiving only the sensitive information required for their operation.

Network Insight · Engineering Lab 07 · Kubernetes Security

Secure Container Deployment Pipeline

Investigate image provenance, vulnerability scanning, software bills of materials, registry integrity, admission verification and secret protection. Inject supply-chain failures and determine whether a container image should be admitted to a production Kubernetes cluster.

Supply-chain investigation

— Signature
— Vulnerability scan
— Secret posture
Ready Deployment verdict
Secure deployment evaluation ready

Select a supply-chain scenario and inspect its deployment security controls.

Supply-chain security controls

Security component inspector

Software Supply-Chain Security

Secure deployment requires more than a successful image build. Provenance, integrity, vulnerability exposure and secret access are separate controls.

Select a pipeline component to inspect its role.
Deployment decision console

Supply-chain engineering findings

Engineering interpretation: Kubernetes does not automatically enforce image signatures, trusted registries or vulnerability thresholds. Those controls require suitable admission policies and supporting tooling. An image signature provides evidence about integrity and signing identity, but does not prove that the software is vulnerability-free. An SBOM is an inventory, not a security verdict. Kubernetes Secrets are not automatically encrypted at rest unless encryption is configured, and encryption at rest does not prevent exposure to an authorised but compromised workload. This widget models policy decisions rather than performing real image verification or encryption.
Network Insight / Kubernetes Security / Section 08

Kubernetes Security Observability and Threat Detection: Audit Logs, Hubble, Runtime Events and SIEM Correlation

Kubernetes security controls can restrict access and reduce attack surfaces, but preventive controls alone cannot guarantee that a cluster will remain uncompromised.

Security teams need visibility into API operations, network communication, container execution, identity activity and changes to cluster configuration. These signals help investigators distinguish legitimate operations from potentially malicious behaviour.

A mature Kubernetes security monitoring architecture combines control-plane audit logs, network flow telemetry, runtime security events, identity signals and centralised event correlation.

8.1 — Kubernetes Security Telemetry Architecture

Security telemetry originates from multiple layers of the Kubernetes environment. Each source provides a different view of workload and cluster behaviour.

KUBERNETES SECURITY OBSERVABILITY Control plane · Network · Runtime · Identity · Correlation API AUDIT HUBBLE RUNTIME IDENTITY API requests Network flows Process events Auth activity EVENT COLLECTION Normalisation / Enrichment / Correlation SIEM / DETECTION Alerts · Investigation · Response MULTI-SOURCE TELEMETRY → CORRELATED SECURITY DETECTION

Architecture interpretation: The diagram represents a logical security monitoring pipeline. Different platforms collect, process and retain telemetry in different ways. Detection quality depends on event coverage, collection reliability, time synchronisation, context enrichment and rule design.

8.2 — Kubernetes API Audit Logging

Kubernetes audit logs record selected API activity according to the configured audit policy.

Audit records can identify the user or service account associated with a request, the requested operation, the affected resource and the resulting response status.

Identity

Which user or service account made the request?

Operation

Which Kubernetes resource or action was requested?

Outcome

Was the operation accepted, rejected or otherwise unsuccessful?

Audit policies determine which events and metadata are captured. Some policies may omit request or response bodies to reduce sensitive data exposure and storage cost.

8.3 — Network Flow Observability with Hubble

Hubble provides network observability for supported Cilium-managed environments.

Flow records can help investigators understand communication between workloads and identify traffic that was permitted or dropped by policy.

Illustrative Hubble Flow Source: namespace: frontend pod: web-7c9d Destination: namespace: database pod: postgres-0 Protocol: TCP Destination Port: 5432 Verdict: DROPPED Potential Interpretation: Unexpected direct database access attempt

Important: A dropped connection does not automatically indicate malicious activity. Misconfiguration, application changes and legitimate troubleshooting can produce similar events. Correlate flow data with workload identity and application context.

8.4 — Container Runtime Security Events

Runtime security monitoring observes selected behaviour of executing containers and host processes.

Depending on the monitoring technology, events may include process execution, file access, privilege changes and suspicious system activity.

Runtime signal Potential significance
Unexpected shell execution May indicate interactive access or application exploitation
Access to sensitive files May indicate credential discovery or configuration inspection
Privilege-related operation May indicate attempted privilege escalation
Unusual outbound connection May indicate command-and-control or unauthorised communication
Unexpected binary execution May indicate malware execution or unauthorised tooling
Container-to-host interaction May indicate a risky configuration or attempted boundary crossing

Runtime events should be evaluated against expected application behaviour and the workload's security configuration.

8.5 — Identity and Authorisation Telemetry

Identity systems provide signals about authentication attempts, token usage, certificate issuance, credential rotation and authorisation decisions.

These signals can help identify unusual service account activity, unexpected identity use or repeated access denials.

Identity Investigation ServiceAccount: payments-api Observed Activity: Repeated API access denials Target Resource: Secrets Expected Permission: No Secret read access Investigation: Determine whether activity is misconfiguration or compromise

Identity telemetry should be correlated with API audit records, workload events and network flows.

8.6 — Security Event Correlation

Individual events often provide incomplete evidence. Correlation combines signals from multiple sources to reconstruct a possible sequence of security-relevant activity.

Example Correlation Sequence 1. Runtime alert: Unexpected shell execution 2. API audit event: ServiceAccount attempts Secret read 3. Hubble flow: Unexpected database connection 4. Identity event: Repeated authorisation failures 5. SIEM correlation: Potential workload compromise

Correlation rules should consider time windows, workload identity, namespace, pod metadata, node information and other available context.

Engineering consideration: Pod names and IP addresses can change as workloads are rescheduled. Stable application labels, workload ownership metadata and identity context can improve event correlation.

8.7 — Kubernetes Security Incident Timeline

The following scenario illustrates how multiple telemetry sources can contribute to a security investigation.

Illustrative Incident Timeline 10:00:00 Application receives suspicious request 10:00:08 Runtime monitor detects shell execution 10:00:15 ServiceAccount attempts Secret access 10:00:18 Kubernetes API returns Forbidden 10:00:24 Hubble records unusual east-west traffic 10:00:30 Network policy drops database connection 10:00:42 SIEM correlates related events 10:01:00 Security team begins investigation

Investigation principle: This is a hypothetical scenario, not evidence of a real incident. A reliable investigation must validate timestamps, event relationships and alternative explanations before concluding that a workload is compromised.

8.8 — SIEM Integration and Detection Engineering

Security Information and Event Management platforms can centralise Kubernetes telemetry alongside cloud, identity, network and endpoint events.

A detection pipeline typically collects events, normalises fields, enriches context, applies detection logic and generates alerts for investigation.

01
Collection

Ingest audit, network and runtime telemetry.

02
Normalisation

Standardise timestamps, identities and event fields.

03
Enrichment

Attach workload, namespace and asset context.

04
Detection

Evaluate rules and behavioural signals.

05
Investigation

Correlate alerts and validate evidence.

06
Response

Contain threats and preserve relevant evidence.

8.9 — Detection Quality and False Positives

A useful detection system must balance visibility with operational accuracy.

Excessive alert volume can overwhelm analysts, while overly restrictive detection logic may miss important activity.

Detection challenge Engineering response
High false-positive rate Improve application baselines and contextual filtering
Missing telemetry Validate collectors, permissions and data pipelines
Unreliable timestamps Synchronise clocks and preserve event timing metadata
Changing pod identities Correlate workload ownership and stable metadata
High storage cost Apply appropriate retention and collection policies
Sensitive log content Restrict access, minimise collection and protect storage

8.10 — Production Security Observability Troubleshooting

Kubernetes Observability Diagnostics kubectl get events -A kubectl get pods -A -o wide kubectl get serviceaccounts -A kubectl get networkpolicy -A kubectl get validatingwebhookconfigurations kubectl get mutatingwebhookconfigurations # Where Cilium / Hubble is installed cilium status hubble status hubble observe hubble observe --verdict DROPPED

API audit logs are typically collected through control-plane audit configurations or managed-platform integrations. They are not necessarily available through kubectl commands.

8.11 — Kubernetes Security Monitoring Checklist

Kubernetes Security Monitoring Checklist [ ] API audit logging configured [ ] Audit policy coverage reviewed [ ] Network flow visibility enabled [ ] Runtime telemetry deployed [ ] Identity events collected [ ] Event timestamps synchronised [ ] Workload metadata enriched [ ] SIEM ingestion validated [ ] Detection rules documented [ ] False-positive rates reviewed [ ] Sensitive log data protected [ ] Retention policies defined [ ] Alert routing tested [ ] Incident timelines validated [ ] Response procedures exercised
Section 8 — Engineering Takeaway

Kubernetes security observability requires visibility across API activity, workload communication, runtime execution, identity behaviour and security policy decisions. Correlating these signals helps security teams identify potential compromises, investigate lateral movement and distinguish malicious activity from configuration or operational failures.

Network Insight · Engineering Lab 08 · Kubernetes Security

Kubernetes Threat Detection Timeline

Investigate suspicious Kubernetes activity across API audit events, container runtime telemetry, network-flow observations and identity signals. Correlate events into an attack timeline and determine whether the available evidence supports a security alert.

Threat detection investigation

0 Visible events
0 Telemetry sources
0% Detection score
Ready Detection status
Threat investigation ready

Select a scenario and advance through the available security telemetry.

Correlated event timeline

Event inspector

Security Event Correlation

A detection engine combines multiple signals to distinguish suspicious behaviour from ordinary cluster activity.

Select a timeline event to inspect its evidence.
Detection console

Threat detection findings

Detection model: This lab uses synthetic telemetry and a simplified correlation score. The score is not a calibrated probability that an attack occurred. Kubernetes API audit logs, runtime sensors, network telemetry and identity systems provide different evidence. No single signal guarantees detection. Real detection depends on sensor coverage, event collection, retention, time synchronisation, detection rules and investigation context. An alert indicates that a rule or correlation threshold was met, not that malicious activity has been conclusively established.
Network Insight / Kubernetes Security / Section 09

Production Kubernetes Security Troubleshooting and Incident Response: Compromised Pods, Lateral Movement, RBAC Failures and Recovery

Kubernetes security incidents frequently involve multiple infrastructure layers. A suspicious network connection may originate from a compromised application, an incorrectly configured NetworkPolicy, excessive RBAC permissions or a workload using unexpected credentials.

Effective incident response requires engineers to understand how these components interact. Investigators must distinguish configuration failures from malicious behaviour while preserving evidence and minimising disruption to legitimate services.

A production-ready response process combines rapid detection, structured investigation, controlled containment, credential protection, evidence preservation, recovery and post-incident hardening.

9.1 — Kubernetes Security Incident Architecture

Consider a hypothetical incident in which a frontend application is suspected of being compromised. The workload begins making unexpected Kubernetes API requests and attempts to communicate directly with a protected database.

KUBERNETES SECURITY INCIDENT INVESTIGATION Detect · Investigate · Contain · Recover KUBERNETES CLUSTER FRONTEND POD API SERVER DATABASE POD Suspected compromise RBAC / Admission Protected workload INVESTIGATE RBAC CHECK POLICY PROTECTED × UNAUTHORISED LATERAL MOVEMENT BLOCKED SECURITY OPERATIONS / INCIDENT RESPONSE Audit Logs · Hubble Flows · Runtime Events · Evidence CORRELATE EVIDENCE BEFORE SELECTING CONTAINMENT ACTIONS

Architecture interpretation: The diagram represents a hypothetical security incident. A blocked connection or rejected API request is not, by itself, proof of compromise. Investigators must correlate runtime activity, workload identity, API audit events, network flows and expected application behaviour.

9.2 — Kubernetes Security Incident Response Lifecycle

A structured response process reduces the risk of destroying evidence, interrupting legitimate workloads or allowing an attacker to continue operating.

01
Detect

Identify suspicious activity through security telemetry.

02
Triage

Assess severity, scope and potential business impact.

03
Investigate

Correlate API, network, runtime and identity evidence.

04
Contain

Restrict suspicious activity while preserving evidence.

05
Recover

Restore trusted workloads and validate security.

06
Improve

Address root causes and strengthen preventive controls.

9.3 — Investigating a Suspected Compromised Pod

The investigation should begin by identifying the affected workload and its operational context.

Relevant information includes the namespace, pod owner, container image digest, service account, worker node, network policies and recent deployment changes.

Initial Workload Investigation kubectl get pods -A -o wide kubectl describe pod \ -n production suspicious-pod kubectl get pod \ -n production suspicious-pod -o yaml kubectl get events \ -n production --sort-by=.lastTimestamp kubectl logs \ -n production suspicious-pod \ --timestamps

These commands provide configuration and operational context but do not establish whether a compromise occurred.

Evidence preservation: Deleting or restarting a suspicious pod may destroy volatile evidence. Before taking disruptive actions, follow the incident-response plan and preserve relevant logs, metadata and forensic information where feasible.

9.4 — Investigating Kubernetes RBAC Failures

Unexpected Kubernetes API requests can indicate misconfiguration, automation errors or possible misuse of workload credentials.

The investigation should identify the request identity, resource, verb, namespace and authorisation result.

RBAC Investigation kubectl auth can-i get secrets \ -n production \ --as=system:serviceaccount:production:app kubectl get roles -A kubectl get rolebindings -A kubectl get clusterroles kubectl get clusterrolebindings

The impersonation command requires appropriate impersonation permissions. API audit records may provide additional evidence about actual requests.

Engineering distinction: An RBAC denial may show that a security boundary worked as intended. It does not necessarily indicate that the requesting workload was malicious.

9.5 — Investigating NetworkPolicy and Cilium Denials

A blocked network connection may be caused by ingress isolation, egress isolation, incorrect labels, missing allow rules or additional Cilium-specific policies.

Investigators should evaluate both communication endpoints and determine which security controls apply.

Network Policy Investigation kubectl get networkpolicy -A kubectl describe networkpolicy \ -n production kubectl get pods \ -n production --show-labels kubectl get namespaces --show-labels # If Cilium / Hubble is available cilium status hubble observe --verdict DROPPED
Observation Potential explanation
Ingress denied Destination workload does not permit the source
Egress denied Source workload does not permit the destination
Unexpected allowed traffic Another additive policy or different enforcement behaviour
DNS resolution failure DNS egress blocked or DNS infrastructure unavailable
Service connectivity failure Policy, service forwarding, endpoint or routing issue

9.6 — Detecting Lateral Movement

Lateral movement occurs when an attacker attempts to extend access from an initially compromised system to other resources.

In Kubernetes, possible targets include internal services, databases, other workloads, cluster APIs and accessible node resources.

Illustrative Lateral Movement Indicators Unexpected source workload | v New internal connection | v Sensitive destination | v Network policy verdict | v Identity and runtime correlation | v Security investigation

Unusual network activity should be compared with known application dependencies, deployment changes and expected communication patterns.

9.7 — Credential Exposure and Service Account Misuse

A compromised workload may expose credentials available to its processes, including projected service account tokens, application secrets or external service credentials.

The potential impact depends on the permissions associated with those credentials.

Identify Exposure

Determine which credentials were accessible to the affected workload.

Assess Permissions

Review Kubernetes RBAC and external system authorisation.

Rotate Credentials

Revoke or rotate affected credentials through approved procedures.

Important: Deleting a pod does not guarantee immediate invalidation of every credential it accessed. Token lifetimes, external identity systems and secret rotation mechanisms must be considered.

9.8 — Kubernetes Incident Containment

Containment actions should reduce attacker access while preserving necessary evidence and avoiding uncontrolled disruption.

Containment action Consideration
Restrict network communication Apply validated policies and confirm effective isolation
Revoke credentials Assess token validity, rotation and external access
Restrict API permissions Review effective RBAC and affected workloads
Isolate affected nodes Consider cluster availability and forensic requirements
Stop compromised workloads Preserve evidence before termination where possible
Block malicious artifacts Prevent redeployment through registry and admission controls

Containment principle: Containment should be proportionate to the confirmed or suspected threat. High-impact actions require coordination with operational and security stakeholders.

9.9 — Evidence Preservation and Forensic Investigation

Kubernetes workloads are ephemeral. Pods may disappear, containers may restart and nodes may be replaced during normal operations.

Incident response procedures should account for this short-lived infrastructure.

Evidence Collection Priorities 1. Preserve relevant API audit records 2. Export workload and namespace metadata 3. Preserve available runtime telemetry 4. Collect network flow records 5. Record image digests and deployment versions 6. Preserve relevant identity events 7. Record timestamps and collection methods 8. Maintain evidence access controls 9. Document containment actions 10. Maintain chain-of-custody records

The exact collection procedure depends on platform capabilities, legal requirements and incident-response policy.

9.10 — Recovery and Security Validation

Recovery should restore trusted application functionality while reducing the likelihood of recurrence.

Rebuilding affected workloads from verified artifacts is often preferable to continuing to operate containers whose integrity cannot be established.

01
Remove Root Cause

Address exploited vulnerabilities or misconfigurations.

02
Restore Trusted State

Deploy verified images and configuration.

03
Validate Security

Test RBAC, network policy and runtime controls.

04
Restore Service

Confirm application health and expected dependencies.

05
Monitor Closely

Watch for recurring indicators and unexpected behaviour.

06
Improve Controls

Update detection, hardening and response procedures.

9.11 — Production Kubernetes Security Incident Checklist

Kubernetes Security Incident Checklist [ ] Incident severity assessed [ ] Affected workloads identified [ ] Namespace and node scope established [ ] API audit evidence preserved [ ] Runtime telemetry collected [ ] Network flows investigated [ ] RBAC permissions reviewed [ ] Service account exposure assessed [ ] Secret exposure assessed [ ] Network policy enforcement validated [ ] Containment actions documented [ ] Credentials rotated where required [ ] Trusted artifacts verified [ ] Workloads safely restored [ ] Application functionality tested [ ] Security controls revalidated [ ] Post-incident review completed
Section 9 — Engineering Takeaway

Production Kubernetes security troubleshooting requires a multi-layer investigation across workload configuration, API access, network enforcement, identity, runtime behaviour and security telemetry. The strongest incident-response processes preserve evidence, validate root causes, contain threats carefully and restore workloads from trusted configurations and artifacts.

Network Insight · Engineering Lab 09 · Kubernetes Security

Kubernetes Security Incident Investigation Lab

Investigate a production Kubernetes security incident. Collect API audit, runtime and network evidence; identify the root cause; select containment measures; and verify whether the cluster has returned to a controlled security posture.

Production incident investigation

0 Evidence collected
— Root cause
— Containment
Active Incident status
Production incident active

Collect evidence before selecting containment and recovery actions.

Evidence collection

Investigation inspector

Incident Response Architecture

Effective incident response requires evidence collection, containment, root-cause analysis, remediation and recovery verification.

Select an architecture component to inspect its operational role.
Investigation console

Incident response assessment

Production engineering guidance: This is a synthetic incident-response exercise. Real Kubernetes investigations should preserve relevant logs, audit events, workload metadata, image digests and forensic evidence before destructive remediation when feasible. Containment actions may interrupt production services, and credential revocation or workload isolation may require several coordinated controls. Recovery verification should include policy checks, workload integrity, identity validation, monitoring and application health. The simulation does not execute Kubernetes commands or modify live infrastructure.
☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Matt Conran
Latest posts by Matt Conran (see all)
Tags: No tags

Comments are closed.