Kubernetes Security Architecture: Protecting Cloud-Native Workloads with Network Policies, Cilium and Zero Trust
Kubernetes provides a powerful platform for deploying, scaling and operating containerised applications. However, the same distributed architecture that makes Kubernetes flexible also introduces a complex security environment involving APIs, worker nodes, pods, network connections, workload identities and software supply chains.
Securing Kubernetes requires more than protecting the cluster perimeter. Modern production environments must control how workloads are deployed, which identities can access cluster resources, how services communicate, and what happens when a container is compromised.
This engineering guide explores how Kubernetes security controls operate together, from API authentication and admission control to network microsegmentation, Cilium eBPF enforcement, workload identity, runtime protection and continuous security observability.
Why Kubernetes Security Requires a Layered Architecture
Kubernetes is not a single security boundary. It is a collection of distributed components, each with different responsibilities and potential attack surfaces.
The Kubernetes API server controls access to cluster resources. Admission mechanisms evaluate requests before supported objects are persisted. Worker nodes execute container workloads. Networking components determine how pods and services communicate. Runtime controls restrict what processes can do after deployment.
A secure architecture must coordinate these layers rather than relying on any single security mechanism.
Control Plane Security
Protect API access, authentication, authorisation, admission policies and audit events.
Network Security
Enforce workload communication boundaries through network policies and microsegmentation.
Workload Identity
Authenticate workloads and authorise service-to-service access using appropriate identities.
Runtime Protection
Reduce container privileges and monitor suspicious runtime behaviour.
Supply Chain Security
Validate container images, dependencies, deployment artifacts and software integrity.
Security Observability
Correlate network flows, audit logs, runtime events and security detections.
Important engineering distinction: Kubernetes NetworkPolicy resources describe network traffic rules, but enforcement depends on the installed networking implementation. Cilium provides an eBPF-based datapath and additional policy capabilities, while workload authentication and application-layer authorisation require appropriate identity and security controls. Network policies alone do not establish a complete Zero Trust architecture.
Security Is an End-to-End Engineering Responsibility
Production Kubernetes environments frequently combine managed cloud infrastructure, Kubernetes control-plane services, container runtimes, third-party networking components, application frameworks and CI/CD pipelines.
Security ownership therefore varies depending on the deployment model. In a managed Kubernetes service, the cloud provider may operate parts of the control plane, while the organisation remains responsible for workload configuration, identities, network policies, application security and other customer-managed controls.
Infrastructure Protection
Protect cluster access, node configuration, network boundaries, secrets and software deployment pipelines.
Workload Protection
Apply least privilege, restrict service communication, validate deployment artifacts and monitor runtime activity.
Zero Trust principle: A workload should not receive unrestricted access merely because it runs inside the same Kubernetes cluster or namespace. Access decisions should be based on verified identity, explicit authorisation, least privilege and relevant security context.
What This Engineering Guide Will Teach
The following nine sections explain how to design, enforce, observe and troubleshoot Kubernetes security in production. We will examine NetworkPolicy, Cilium, eBPF, Zero Trust, RBAC, admission control, runtime security, secrets, software supply chains and incident response, with an emphasis on real traffic behaviour, engineering trade-offs and operational diagnostics.
Kubernetes Security Architecture
Explore nine interconnected security layers, from workload isolation and network enforcement to Zero Trust identity, secrets protection, runtime detection and production incident response. Select a component to explore its engineering section.
Security Architecture Fundamentals
Understand Kubernetes trust boundaries, control-plane security, worker-node protection and defence in depth.
Kubernetes Security Architecture Fundamentals: Control Plane, Worker Nodes, Pods and Trust Boundaries
Kubernetes security begins with understanding the architecture of the cluster itself. A Kubernetes environment contains several distinct security domains, including the control plane, worker nodes, container workloads, networking components, storage systems and external management interfaces.
Each domain introduces different attack surfaces and enforcement mechanisms. A secure deployment must protect the interfaces between these components while limiting the privileges available to users, services and applications.
1.1 — Kubernetes Security Architecture
The Kubernetes control plane manages cluster state and workload orchestration. Worker nodes execute containers and provide the networking and runtime environment required by applications.
Security interpretation: The API server is the primary control-plane entry point for Kubernetes API operations. Authentication, authorisation and admission control govern requests, while worker-node security and networking controls protect workload execution and communication. These are separate enforcement boundaries that must be secured together.
1.2 — Kubernetes API Request Security
A Kubernetes API request passes through multiple security stages before the requested operation is completed.
Authentication establishes the identity associated with the request. Authorisation determines whether that identity is permitted to perform the requested operation. Admission control can validate or modify supported API requests according to configured policies.
Important: Admission control is not a replacement for authentication or authorisation. It provides an additional policy enforcement stage for applicable API operations. Some operations and API paths have different admission behaviour.
1.3 — Control Plane Security Components
| Component | Security responsibility |
|---|---|
| API server | Secures Kubernetes API access and request processing |
| Authentication | Establishes user or workload identity |
| RBAC | Controls permitted Kubernetes API actions |
| Admission control | Validates or modifies supported resource requests |
| etcd | Stores cluster state and requires strong access protection |
| Controller manager | Reconciles cluster resources according to desired state |
| Scheduler | Assigns eligible workloads to worker nodes |
1.4 — Worker Node and Container Security
Worker nodes provide the execution environment for Kubernetes workloads. They typically run the kubelet, container runtime and networking components required to operate pods.
Compromise of a worker node may expose multiple workloads running on that node. Security controls should therefore protect the host operating system, container runtime, kubelet API, credentials and privileged processes.
Node Hardening
Minimise services, patch the operating system and restrict administrative access.
Kubelet Security
Protect kubelet authentication, authorisation and network exposure.
Container Isolation
Use appropriate security contexts, capabilities, seccomp and runtime controls.
1.5 — Pod Security Boundaries
Pods are Kubernetes deployment and scheduling units, but they should not be treated as strong security boundaries equivalent to separate virtual machines.
Containers within a pod share a network namespace and may share storage volumes and other resources. The level of isolation between pods depends on the operating system, container runtime, security configuration and workload privileges.
Engineering implication: A compromised container may attempt to access shared pod resources, other workloads, node services or Kubernetes credentials. Pod security settings and network segmentation reduce the available attack surface but do not eliminate every possible escape path.
1.6 — Namespaces and Multi-Tenancy
Namespaces organise Kubernetes resources and provide scopes for many API objects. They can support administrative separation, resource policies and network policy selection.
However, namespaces alone do not provide complete security isolation. Multi-tenant environments require additional controls such as RBAC, network policies, admission policies, resource quotas, workload identity and node isolation where appropriate.
| Control | Purpose |
|---|---|
| Namespace | Logical resource organisation and policy scope |
| RBAC | API permissions for users and service accounts |
| NetworkPolicy | Restricts selected ingress and egress traffic |
| Pod Security Admission | Enforces configured pod security standards |
| ResourceQuota | Limits resource consumption within a namespace |
| Node isolation | Separates sensitive workloads at the infrastructure level |
1.7 — Kubernetes Shared Responsibility
Security responsibilities differ between self-managed Kubernetes clusters and managed services such as AKS, EKS and GKE.
Platform Provider
Depending on the service model, the provider may manage control-plane infrastructure, service availability, platform updates and selected underlying infrastructure controls.
Cluster Operator
The organisation remains responsible for its workload security, identities, network configuration, application permissions, secrets and other customer-managed controls.
The exact division of responsibility must be verified against the chosen Kubernetes service and deployment model.
1.8 — Defence-in-Depth Architecture
Defence in depth combines multiple independent security controls so that failure of one mechanism does not automatically expose the entire cluster.
Zero Trust alignment: Defence in depth supports Zero Trust by reducing implicit trust and enforcing explicit access decisions. However, a complete Zero Trust architecture also requires verified identities, appropriate authorisation and continuous evaluation of security context.
1.9 — Security Architecture Validation Checklist
The checklist provides a starting point for engineering validation. Production security requirements should also consider regulatory obligations, threat models, deployment architecture and organisation-specific risk.
Section 1 — Engineering Takeaway
Kubernetes security is a layered architecture spanning the control plane, API request pipeline, worker nodes, container runtime, workloads, network communication and identity systems. Understanding these trust boundaries is essential before implementing network microsegmentation, Cilium enforcement or Zero Trust policies.
Kubernetes Security Architecture Explorer
Explore Kubernetes trust boundaries, control-plane security, worker-node protection, workload isolation, network enforcement and defence in depth. Select security layers, inspect architecture components and investigate how architectural controls affect common Kubernetes attack scenarios.
Cluster security architecture
Defence-in-depth controls
Architecture inspector
Kubernetes Trust Boundaries
Kubernetes security depends on multiple independent controls protecting the control plane, worker nodes, workloads and network paths.
Select a component in the diagram to inspect its security responsibility.Security investigation console
Architecture findings and engineering recommendations
Kubernetes Network Policies and Microsegmentation: Ingress, Egress, Default-Deny and Traffic Enforcement
Kubernetes networking is designed to allow workloads to communicate across a cluster. In a conventional deployment without enforced network isolation, pods can often reach other pods regardless of their namespace or the worker node on which they run.
This connectivity simplifies application deployment, but it also creates an opportunity for lateral movement if a workload becomes compromised. An attacker who gains access to one container may attempt to reach databases, internal APIs, administrative endpoints or other services.
Kubernetes NetworkPolicy provides a declarative mechanism for controlling selected ingress and egress traffic. When supported and enforced by the cluster networking implementation, policies can restrict communication to explicitly approved sources, destinations and ports.
2.1 — Kubernetes Microsegmentation Architecture
Microsegmentation divides the cluster into smaller communication domains. Instead of assuming that all internal workloads are trusted, engineers define which application components require connectivity and which communication paths should remain blocked.
Architecture interpretation: The diagram represents an intended policy outcome: frontend workloads can reach the API, API workloads can reach the database, and direct frontend-to-database communication is blocked. In a real deployment, both ingress and egress isolation must be evaluated, and the installed CNI must support NetworkPolicy enforcement. Namespace boundaries alone do not create these restrictions.
2.2 — How Kubernetes NetworkPolicy Works
A NetworkPolicy is a namespaced Kubernetes resource. Its podSelector identifies the pods to which the policy applies. The policyTypes field identifies whether the policy controls ingress, egress or both.
For an isolated direction, traffic is allowed only when it matches an applicable policy rule, subject to the behaviour of the networking implementation and traffic outside the NetworkPolicy model.
If a source pod is not isolated for egress, its egress traffic is permitted by NetworkPolicy unless another control blocks it. Likewise, a destination pod that is not isolated for ingress accepts traffic unless another control prevents it.
Important: Kubernetes NetworkPolicies are additive. Standard NetworkPolicy does not provide an explicit deny rule that overrides an allow. Multiple applicable policies contribute to the set of permitted traffic.
2.3 — Default-Deny Network Policies
A default-deny policy establishes a restrictive starting point by selecting pods and isolating their traffic. Engineers then add narrowly scoped allow policies for required communication.
Default-Deny Ingress and Egress
This policy selects all pods in the app namespace and isolates both ingress and egress. Because no allow rules are specified, it permits no ordinary ingress or egress traffic under standard NetworkPolicy semantics.
However, some traffic may remain outside NetworkPolicy enforcement, including certain node-related communication. Behaviour must be validated against the CNI and platform.
Production consideration: Applying default-deny egress without allowing DNS, required APIs, monitoring or external dependencies can break applications. Build and validate the required communication inventory before enforcing restrictive policies.
2.4 — Pod Selectors and Namespace Selectors
NetworkPolicy rules use label selectors to identify which peers are permitted. These selectors can match pods within a namespace or combine namespace and pod labels to select workloads across namespaces.
| Selector | Purpose | Engineering example |
|---|---|---|
| podSelector | Select pods by label | Allow app=frontend |
| namespaceSelector | Select namespaces by label | Allow environment=production |
| Combined selectors | Select specific pods within selected namespaces | Allow app=api in team=payments namespaces |
| ipBlock | Select an IP CIDR range | Allow approved external network ranges |
| ports | Limit allowed transport ports and protocols | Allow TCP 443 |
Selector distinction: A single peer entry containing both namespaceSelector and podSelector matches pods satisfying both conditions. Separate peer entries are alternatives. Confusing these structures can unintentionally broaden access.
2.5 — Allow Frontend Traffic to the API
The following policy permits TCP port 8080 ingress to API pods in the app namespace from frontend pods in namespaces labelled role=web.
This policy permits the specified ingress traffic. It does not automatically grant the frontend pod egress permission if that pod is already isolated for egress. A corresponding egress allow rule may therefore be required.
2.6 — DNS Egress and Essential Dependencies
DNS is a common source of application failures after default-deny egress is enabled. Workloads may be unable to resolve Kubernetes Service names or external dependencies even though their application code and service configuration are otherwise correct.
A production policy should identify the actual DNS service, namespace labels, pod labels and transport requirements rather than assuming a universal CoreDNS deployment configuration.
Engineering note: Standard Kubernetes NetworkPolicy does not provide DNS-name-based egress filtering. Some CNIs, including Cilium, offer additional FQDN-aware policy capabilities through their own policy resources.
2.7 — NetworkPolicy Enforcement and the CNI
Kubernetes stores NetworkPolicy objects through its API, but the API server does not itself filter application packets. Actual traffic enforcement is performed by a compatible networking implementation.
Kubernetes API
Stores the declarative policy configuration, including selectors, directions, peers and ports.
Network Enforcement
The installed CNI or associated datapath translates supported policy rules into packet-forwarding decisions.
Implementations may use Linux networking facilities, eBPF or other platform-specific mechanisms. Capabilities and treatment of special traffic cases differ between implementations.
2.8 — Production Microsegmentation Design
Effective microsegmentation begins with application communication discovery. Engineers must identify legitimate service dependencies before constructing policies.
Discover Traffic
Map application flows, service dependencies and required external endpoints.
Classify Workloads
Use consistent labels for applications, environments and security domains.
Define Trust Boundaries
Separate frontend, application, data and management traffic.
Apply Default Deny
Introduce isolation with staged testing and operational safeguards.
Allow Required Flows
Grant only the sources, destinations and ports needed.
Validate Enforcement
Test permitted and denied paths and monitor application health.
2.9 — NetworkPolicy Troubleshooting
| Symptom | Potential cause | Investigation |
|---|---|---|
| Pod cannot reach API | Missing egress or ingress allow rule | Check policies selecting both endpoints |
| DNS resolution fails | DNS traffic blocked by egress isolation | Inspect DNS flow and allow rules |
| Unexpected access remains possible | Another additive allow policy or unsupported traffic case | Review all applicable policies and CNI behaviour |
| Namespace selector does not match | Incorrect namespace labels | Inspect namespace metadata |
| Policy exists but is ineffective | CNI lacks required enforcement capability | Verify network plugin support and health |
| Service traffic behaves unexpectedly | Address translation or implementation-specific processing | Trace service forwarding and policy enforcement points |
When Cilium is installed, Hubble flow visibility can provide additional information about forwarded and dropped traffic. Exact commands depend on the installed version and tooling.
2.10 — Engineering Security Checklist
Section 2 — Engineering Takeaway
Kubernetes NetworkPolicy provides a foundational mechanism for network microsegmentation, ingress and egress isolation, least-privilege connectivity and reduced lateral-movement opportunities. Effective deployment requires accurate workload labels, a compatible enforcement implementation, complete application dependency mapping and continuous validation of both permitted and denied traffic.
Network Policy Traffic Enforcement Simulator
Build a Kubernetes microsegmentation policy environment. Select source and destination pods, test TCP or UDP traffic, inspect namespace and label selectors, and observe how ingress and egress isolation combine to permit or deny a connection.
Traffic evaluation workspace
NetworkPolicy configuration
Enable or disable the policy objects below. Policies are additive: multiple policies can select the same pod without rule-order precedence.
Policy inspector
NetworkPolicy Evaluation
NetworkPolicy rules select destination or source pods using labels and namespaces.
A connection must satisfy the applicable source egress and destination ingress requirements.Policy decision console
Engineering findings
Cilium and eBPF Security Architecture: Kernel Enforcement, Workload Identity, L3–L7 Policy and Hubble Observability
Cilium is a cloud-native networking, security and observability platform that uses extended Berkeley Packet Filter (eBPF) technology to implement networking functions and enforce security policies within supported Linux environments.
Rather than relying exclusively on traditional packet-filtering chains, Cilium can attach eBPF programs to supported kernel networking hooks, allowing packet processing and policy decisions to occur at specific points in the Linux networking datapath.
A major architectural feature is Cilium's use of security identities derived from workload labels. This allows policies to express relationships between workloads without depending solely on their changing IP addresses.
3.1 — Cilium eBPF Security Datapath Architecture
Cilium separates policy configuration from packet enforcement. Kubernetes resources and Cilium policy objects express desired security behaviour. The Cilium agent processes this configuration and programs the datapath accordingly.
Architecture interpretation: The diagram shows a logical enforcement flow. Actual Cilium packet processing depends on the endpoint type, kernel capabilities, routing configuration, selected datapath features and policy type. eBPF programs may execute at different kernel hooks, and some L7 policies involve proxy processing.
3.2 — What eBPF Changes in Kubernetes Networking
eBPF enables verified programs to run at supported points within the Linux kernel. These programs can inspect packet metadata, perform lookups in kernel maps and apply networking decisions.
Cilium uses eBPF to implement supported networking, load-balancing, security and observability functions.
Kernel Enforcement
Policy decisions can occur in supported kernel datapath hooks.
Identity-Based Security
Workloads can be represented by label-derived security identities.
Policy Maps
Kernel maps store data used by eBPF programs for enforcement.
Flow Visibility
Network events can be exposed through Cilium and Hubble.
Service Processing
eBPF can support Kubernetes service load-balancing functions.
Extended Policy
Cilium adds capabilities beyond standard Kubernetes NetworkPolicy.
3.3 — Cilium Security Identities
Cilium assigns security identities to endpoints based on selected security-relevant labels. Endpoints sharing an equivalent security label set may share the same identity.
This approach allows policies to express relationships between workload categories rather than relying exclusively on individual pod IP addresses.
Important: The identity numbers above are illustrative. Cilium identity allocation and label handling depend on the environment and configuration. An identity is not equivalent to a Kubernetes ServiceAccount identity or a cryptographic workload identity.
3.4 — Cilium Policy Enforcement and eBPF Maps
Cilium agents translate supported policies into datapath state. eBPF programs consult kernel maps containing relevant endpoint, identity, connection and policy information.
The exact lookup sequence varies by traffic direction, endpoint type and enabled features. Not every packet necessarily follows an identical chain of operations.
3.5 — Layer 3, Layer 4 and Layer 7 Security
| Layer | Policy capability | Example |
|---|---|---|
| Layer 3 | Workload identity, endpoint or network-based controls | Frontend can reach API workloads |
| Layer 4 | Protocol and transport port restrictions | Allow TCP port 8080 |
| Layer 7 | Supported application protocol rules | Allow selected HTTP methods and paths |
Layer 7 policy requires protocol-aware inspection. Depending on the policy and traffic type, Cilium can integrate with Envoy proxy processing to enforce application-layer rules.
Encryption limitation: Application-layer inspection of encrypted traffic requires an appropriate mechanism for accessing the relevant application data. Cilium cannot inspect arbitrary encrypted HTTP payloads simply because an eBPF program observes the packets.
3.6 — CiliumNetworkPolicy and Extended Security Controls
Cilium supports standard Kubernetes NetworkPolicy resources and its own policy APIs, including CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy.
These additional resources provide Cilium-specific capabilities such as supported L7 rules, identity-aware selectors and selected DNS-aware egress controls.
This example illustrates an HTTP policy structure. Cross-namespace selection, label scoping and deployment behaviour must be verified against the installed Cilium version and policy documentation.
3.7 — Hubble Network Observability
Hubble provides network flow visibility for Cilium-managed environments. It can expose information about endpoints, identities, traffic directions, policy verdicts and supported protocol details.
Flow Records
Observe network traffic between workloads and endpoints.
Policy Verdicts
Investigate forwarded and dropped communication.
Service Visibility
Relate observed traffic to Kubernetes workloads and services.
Observability distinction: Hubble visibility depends on the configured deployment, supported protocols, available flow events and telemetry settings. It should not be treated as a complete packet-capture replacement for every troubleshooting case.
3.8 — Production Cilium Troubleshooting
| Symptom | Potential cause | Investigation |
|---|---|---|
| Traffic unexpectedly denied | Policy selector or identity mismatch | Inspect endpoint identity and policy verdict |
| Policy not enforced | Configuration or datapath issue | Verify agent health and policy state |
| Service connectivity failure | Service translation, routing or policy issue | Trace service and backend traffic |
| Unexpected L7 rejection | HTTP rule or proxy processing issue | Inspect L7 policy and proxy events |
| Missing flow visibility | Hubble configuration or event limitations | Verify Hubble status and collection settings |
| Node-to-node communication failure | Routing, tunnelling, MTU or firewall issue | Inspect datapath and underlay connectivity |
These commands assume the corresponding CLI tools are installed and configured. Available flags and behaviour vary by Cilium and Hubble version.
3.9 — Cilium Security Engineering Checklist
Section 3 — Engineering Takeaway
Cilium extends Kubernetes network security through eBPF-based datapath enforcement, label-derived security identities, L3–L7 policy capabilities and Hubble observability. Its value comes from connecting declarative workload policy with efficient kernel-level network processing and actionable traffic visibility. Successful production deployment requires careful identity design, policy validation and datapath troubleshooting.
Cilium eBPF Packet Inspection Visualiser
Follow a packet through a simplified Cilium datapath. Inspect endpoint security identities, eBPF policy lookups, optional L7 proxy enforcement, forwarding decisions and Hubble-style flow observations.
Packet inspection configuration
Datapath ready
Inspect a packet to construct the datapath stages.
eBPF policy map inspector
Datapath component inspector
Cilium Security Architecture
Cilium associates endpoint security identities with workload labels and evaluates policy using its datapath and supporting components.
Select a component in the diagram to inspect its role.Hubble-style flow console
Packet inspection findings
Zero Trust Security for Kubernetes: Workload Identity, mTLS, Least Privilege and Service-to-Service Authorisation
Zero Trust is a security architecture based on the principle that access should not be granted solely because a user, workload or network connection originates from a trusted location. Every access request should be evaluated against explicit identity and authorisation requirements.
In Kubernetes, workloads are dynamic. Pods are created, terminated and rescheduled across worker nodes. Their IP addresses may change frequently, and services communicate across namespaces and cluster boundaries.
A production Zero Trust architecture therefore requires controls that can identify workloads, authenticate communication peers, enforce least-privilege access and provide visibility into service-to-service communication.
4.1 — Kubernetes Zero Trust Architecture
A common Zero Trust design combines workload identities, encrypted communication, service authorisation, network segmentation and security observability.
Architecture interpretation: The diagram illustrates a logical Zero Trust service communication model. Identity issuance, certificate distribution, mTLS termination and authorisation enforcement depend on the selected implementation. SPIFFE/SPIRE, service meshes and cloud-native workload identity systems provide different capabilities and integration models.
4.2 — Zero Trust Principles in Kubernetes
Verify Identity
Authenticate workloads using appropriate identities and trusted credentials.
Least Privilege
Permit only the API actions and service communication required by each workload.
Explicit Authorisation
Evaluate requests against defined security policies.
Encrypted Communication
Protect sensitive service traffic using appropriate transport security.
Segmentation
Restrict unnecessary east-west and north-south communication.
Continuous Visibility
Monitor identity, network flows and authorisation decisions.
4.3 — Kubernetes Service Accounts and Workload Identity
Kubernetes ServiceAccounts provide identities for processes running in pods when interacting with the Kubernetes API and compatible integrated systems.
Modern Kubernetes deployments can use projected, bounded service account tokens that have configurable audiences and expiration characteristics.
These tokens can support workload identity federation with external systems when the receiving platform is configured to trust the appropriate issuer and validate the token.
Security distinction: A Kubernetes ServiceAccount token is not automatically an mTLS certificate. Service account authentication, federated cloud identity and SPIFFE-based X.509 identities are related but distinct mechanisms.
4.4 — SPIFFE and SPIRE Architecture
SPIFFE defines a framework for workload identity across heterogeneous systems. Workloads can receive SPIFFE identities represented by SPIFFE IDs.
SPIRE is an implementation of SPIFFE that provides workload attestation and identity issuance capabilities.
SPIFFE Verifiable Identity Documents (SVIDs) can be used by supported applications and infrastructure to authenticate workloads.
Engineering consideration: The SPIFFE ID above is illustrative. Actual identity naming and attestation policies depend on the deployment. Secure identity issuance requires trusted workload and node attestation, protected signing infrastructure and appropriate credential rotation.
4.5 — Mutual TLS for Service-to-Service Communication
Mutual TLS allows both communication peers to authenticate each other using certificates during the TLS handshake.
When correctly configured, mTLS provides encrypted transport, peer authentication and protection against certain network-level impersonation and interception attacks.
The exact message sequence depends on the TLS version and authentication configuration. TLS 1.3, for example, uses a different handshake structure from earlier TLS versions.
Important: mTLS authenticates communication peers, but does not by itself decide which application operations those peers may perform. Authorisation policies must still determine whether an authenticated identity can access a particular service or resource.
4.6 — Service Mesh and Zero Trust Enforcement
Service mesh platforms can provide service-to-service security capabilities such as mTLS, workload authentication, traffic policy and telemetry.
Different implementations use different datapath architectures. Some rely on sidecar proxies, while others support alternative proxy placement or ambient models.
| Security function | Purpose |
|---|---|
| Workload authentication | Verify the identity of service communication peers |
| mTLS | Encrypt traffic and authenticate peers |
| Authorisation policy | Permit or deny selected service requests |
| Telemetry | Observe service communication and policy outcomes |
| Certificate management | Issue, distribute and rotate workload credentials |
A service mesh is not mandatory for every Zero Trust deployment. Applications and platform components may implement equivalent controls through other mechanisms.
4.7 — Identity-Aware Authorisation
An authenticated workload should receive only the permissions required to perform its legitimate function.
For example, a frontend service may be permitted to invoke selected API endpoints while being denied access to administrative operations.
This is a conceptual policy example, not a Kubernetes or service mesh configuration file.
4.8 — Continuous Verification and Credential Rotation
Zero Trust architecture requires security controls to remain effective throughout the workload lifecycle.
Workload credentials should be short-lived where practical, rotated securely and revoked or invalidated when their associated trust conditions no longer apply.
Identity Lifecycle
Issue credentials to validated workloads and manage their expiration.
Policy Lifecycle
Review access rules as applications and dependencies change.
Telemetry Lifecycle
Monitor authentication failures, denied requests and suspicious behaviour.
Operational distinction: Continuous verification does not necessarily mean performing a new cryptographic handshake for every application request. It means maintaining appropriate identity, authorisation and monitoring controls throughout the communication lifecycle.
4.9 — Production Zero Trust Troubleshooting
| Symptom | Potential cause | Investigation |
|---|---|---|
| mTLS handshake failure | Certificate, trust or protocol mismatch | Inspect certificate validity and trust configuration |
| Workload identity rejected | Incorrect issuer, audience or identity mapping | Validate token or SVID attributes |
| Service request denied | Authorisation policy mismatch | Inspect caller identity and policy decision |
| Unexpected service access | Overly broad authorisation or missing enforcement | Review effective policies and traffic paths |
| Credential rotation failure | Identity distribution or renewal issue | Inspect identity service and workload logs |
| Service connectivity failure | Network policy, routing or mTLS configuration | Separate network reachability from authentication |
4.10 — Zero Trust Engineering Checklist
Section 4 — Engineering Takeaway
Zero Trust Kubernetes security combines verified workload identities, authenticated communication, encrypted transport, explicit authorisation, least-privilege access and continuous security visibility. Network location alone should not determine trust. Effective implementation requires coordinated identity, networking and application-layer security controls.
Zero Trust Workload Access Decision Lab
Investigate how Kubernetes workloads establish trust across service boundaries. Evaluate service identity, certificate validity, mTLS, authorisation policy, request scope and continuous verification to determine whether service-to-service access should be permitted.
Workload access investigation
Zero Trust verification ready
Select a scenario and evaluate its access decision.
Trust boundary inspector
Workload Trust Architecture
Zero Trust requires explicit authentication, authorisation and ongoing validation rather than trusting traffic solely because it originates inside the cluster.
Select an architecture component to inspect its role.Workload identity
Access decision console
Zero Trust engineering findings
Kubernetes RBAC, Service Accounts and API Security: Authentication, Authorisation, Admission and Privilege Escalation
The Kubernetes API server is one of the most security-critical components in a cluster. It provides the primary interface through which users, controllers, automation systems and workloads interact with Kubernetes resources.
A compromised identity with excessive API permissions may be able to read sensitive configuration, modify workloads, create privileged pods or alter cluster security controls.
Production API security therefore requires strong authentication, least-privilege authorisation, carefully controlled service accounts, admission policies, audit logging and continuous permission review.
5.1 — Kubernetes API Security Decision Pipeline
A Kubernetes API request passes through a sequence of security and validation stages. Each stage performs a different function and may reject the request before it reaches the requested operation.
Architecture interpretation: The diagram is a simplified representation of API request processing. Kubernetes also performs request parsing, validation and other processing steps. Admission applies to supported operations, and read-only requests do not follow the same admission path as resource creation or modification.
5.2 — Kubernetes Authentication
Authentication determines the identity associated with an API request. Kubernetes supports several authentication mechanisms, depending on cluster configuration.
Client Certificates
Authenticate clients using trusted X.509 certificates.
Bearer Tokens
Authenticate supported service account or external identity tokens.
OIDC Integration
Integrate Kubernetes API authentication with a trusted identity provider.
Authentication establishes who the request represents. It does not automatically grant permission to perform an operation.
5.3 — Kubernetes RBAC Architecture
Role-Based Access Control (RBAC) is a Kubernetes authorisation mechanism that associates permissions with users, groups and service accounts.
RBAC rules describe allowed API operations using verbs, API groups, resources and optional resource names.
| RBAC resource | Scope | Purpose |
|---|---|---|
| Role | Namespace | Defines permissions within a namespace |
| ClusterRole | Cluster-wide definition | Defines permissions for cluster-scoped or namespaced resources |
| RoleBinding | Namespace | Grants a Role or ClusterRole within one namespace |
| ClusterRoleBinding | Cluster | Grants a ClusterRole across applicable cluster scope |
Important: A ClusterRole does not automatically grant cluster-wide access. The binding determines where its permissions apply. A RoleBinding can reference a ClusterRole while granting namespaced permissions only within the RoleBinding's namespace.
5.4 — Least-Privilege RBAC Example
Consider an application controller that needs to read pods within the production namespace but does not need to create, modify or delete them.
This Role defines read-only permissions for pods within the production namespace.
The RoleBinding grants those permissions to the specified service account.
Engineering implication: The service account can perform the permitted pod read operations within the namespace, assuming no other authorisation mechanism rejects the request. The binding does not itself grant permission to modify or delete pods.
5.5 — Service Account Token Security
Service accounts allow workloads to authenticate to the Kubernetes API and compatible integrated systems.
Modern Kubernetes deployments typically use projected, time-bound service account tokens rather than automatically creating long-lived token Secrets.
Workloads that do not need Kubernetes API access should avoid unnecessary service account token exposure.
Security consideration: Disabling automatic token mounting reduces one credential exposure path, but it does not prevent all forms of Kubernetes API access. Other credentials or network access may still exist.
5.6 — Admission Controllers and Policy Enforcement
Admission controllers provide additional control over supported API operations after authentication and authorisation.
They can validate requested resources or modify them according to configured platform policies.
Mutating Admission
May modify supported resource objects before they are persisted.
Validating Admission
May accept or reject supported requests based on configured requirements.
Examples include Pod Security Admission, ValidatingAdmissionPolicy and webhook-based admission controls. Availability and capabilities depend on Kubernetes version and configuration.
5.7 — RBAC Privilege Escalation Risks
Overly broad Kubernetes permissions can allow an attacker or compromised workload to gain access beyond its intended responsibilities.
| Permission | Potential security impact |
|---|---|
| Read Secrets | May expose credentials or sensitive application data |
| Create Pods | May allow workloads with dangerous configurations if admission controls permit them |
| Update Deployments | May allow modification of application containers or configuration |
| Create RoleBindings | Can expand access within Kubernetes RBAC escalation restrictions |
| Impersonate | May allow requests to be evaluated as another identity |
| Bind / Escalate | Special permissions that can bypass selected RBAC escalation safeguards |
RBAC escalation protection: Kubernetes includes restrictions designed to prevent users from granting permissions they do not already possess. Special permissions such as bind and escalate can affect these restrictions. Effective privilege depends on the complete set of authorisation rules and admission controls.
5.8 — Kubernetes API Audit Logging
Kubernetes audit logging records selected API activity according to configured audit policies.
Audit records can help security teams investigate changes to resources, access attempts and administrative operations.
Who
Identify the user or service account associated with a request.
What
Record the requested operation and affected resource.
Outcome
Inspect response status, timestamps and available request metadata.
Operational consideration: Audit log content and volume depend on the configured audit policy and backend. Sensitive data exposure, retention, access control and storage capacity must be considered.
5.9 — Production RBAC and API Troubleshooting
| Symptom | Potential cause | Investigation |
|---|---|---|
| HTTP 401 Unauthorized | Authentication failure or missing credentials | Inspect authentication configuration and token validity |
| HTTP 403 Forbidden | Insufficient authorisation or admission rejection | Inspect response message, RBAC and admission policies |
| Service account cannot list pods | Missing RoleBinding or incorrect namespace | Review Role and RoleBinding resources |
| Unexpected cluster-wide access | Broad ClusterRoleBinding or additional permissions | Review effective RBAC bindings |
| Pod creation rejected | Admission policy or resource validation | Inspect API response and admission events |
| Token authentication fails | Expired token, issuer or audience mismatch | Validate token properties and authentication configuration |
The impersonation example requires appropriate permission to impersonate the specified service account. A successful can-i check does not replace testing the complete request path, including admission controls.
5.10 — Kubernetes API Security Checklist
Section 5 — Engineering Takeaway
Kubernetes API security depends on strong authentication, least-privilege RBAC, carefully managed service accounts, admission enforcement and comprehensive audit visibility. The most important engineering objective is to ensure that every identity receives only the permissions necessary for its legitimate operational role.
Kubernetes RBAC Permission Investigation Lab
Investigate Kubernetes API authorisation. Select service accounts, namespaces, API verbs and resources; inspect RoleBindings and ClusterRoleBindings; and determine whether a request is permitted by the effective RBAC rules.
API authorisation investigation
RBAC evaluation ready
Select an API request and evaluate its permissions.
RBAC binding configuration
Authorisation inspector
Kubernetes API Authorisation
Kubernetes RBAC evaluates requests using the permissions granted through RoleBindings and ClusterRoleBindings.
Select a component to inspect its role.API decision console
RBAC engineering findings
Pod Security, Container Isolation and Runtime Protection: Kubernetes Admission, Linux Security and Container Escape Prevention
Kubernetes containers execute application processes using operating-system isolation mechanisms. Unlike conventional virtual machines, standard Linux containers running on the same worker node share the host kernel.
This architecture provides efficient resource utilisation and rapid workload deployment, but it also introduces security considerations. A vulnerable application, excessive Linux capabilities, privileged container or exposed host resource may create opportunities for privilege escalation or movement beyond the intended container boundary.
A production container security architecture therefore combines admission-time configuration controls, Linux process isolation, restricted privileges, hardened container runtimes and continuous runtime monitoring.
6.1 — Kubernetes Container Security Architecture
Container security spans several layers. Kubernetes admission policies govern supported workload configurations before execution. The container runtime creates the workload environment, while Linux kernel mechanisms enforce process, resource and access restrictions.
Architecture interpretation: The diagram represents standard Linux-container isolation. Sandboxed runtimes may introduce additional isolation mechanisms, including virtualisation boundaries. Admission controls operate before supported pod configurations are accepted, while runtime controls protect executing processes.
6.2 — Kubernetes Pod Security Standards
Kubernetes defines Pod Security Standards that describe different levels of workload security restrictions.
| Profile | Security posture | Typical use |
|---|---|---|
| Privileged | Unrestricted profile with minimal policy limitations | Trusted infrastructure workloads requiring elevated access |
| Baseline | Prevents selected known privilege escalation risks | General workloads requiring baseline safeguards |
| Restricted | Applies stronger hardening requirements | Security-sensitive application workloads |
Pod Security Admission can enforce, audit or warn about violations according to namespace labels and configured policy versions.
This example configures the namespace to apply the Restricted profile for the selected admission modes. Actual requirements depend on the Kubernetes version and the applicable Pod Security Standards definition.
6.3 — Linux Namespaces and Process Isolation
Linux namespaces provide isolation for selected operating-system resources. Container runtimes use these mechanisms to create separate process environments.
PID Namespace
Controls process identifier visibility and process namespace isolation.
Network Namespace
Provides separate network interfaces, routing tables and related networking state.
Mount Namespace
Controls visibility of filesystem mount points.
IPC Namespace
Separates selected inter-process communication resources.
UTS Namespace
Provides isolation for hostname and domain-name information.
User Namespace
Supports mapping user and group identifiers between namespaces.
Important: Containers in the same Kubernetes pod normally share a network namespace. Other namespace-sharing behaviour depends on pod settings and runtime configuration. Namespace isolation does not remove the shared-kernel security dependency.
6.4 — Linux Capabilities and Privilege Reduction
Linux capabilities divide selected superuser privileges into more granular permissions.
A container should receive only the capabilities required for its application function. Unnecessary capabilities increase the potential impact of a compromised process.
This configuration requests several common hardening controls. The container image must support running without root privileges and without additional Linux capabilities.
Engineering consideration: Dropping all capabilities and disabling privilege escalation are useful safeguards, but they do not eliminate every container escape or kernel vulnerability. Additional host, runtime and kernel security controls remain necessary.
6.5 — Seccomp System Call Filtering
Seccomp allows Linux processes to operate under restrictions on available system calls.
A seccomp profile can reduce the system call interface available to a container, limiting access to selected kernel functionality.
Kubernetes supports specifying seccomp profiles through the securityContext configuration.
Important: RuntimeDefault behaviour depends on the installed container runtime. A profile may block application operations that require restricted system calls, so compatibility testing is essential.
6.6 — AppArmor and SELinux
Linux Mandatory Access Control mechanisms provide additional restrictions on process behaviour.
AppArmor
Applies profile-based restrictions to supported processes, including controls over filesystem and other operations.
SELinux
Uses security labels and policy rules to enforce Mandatory Access Control decisions.
The available configuration and enforcement behaviour depend on the host operating system, kernel configuration, container runtime and Kubernetes integration.
6.7 — Privileged Containers and Host Access
Privileged containers can receive extensive access to host resources and may bypass important container isolation restrictions.
Other high-risk configurations include unrestricted hostPath mounts, host networking, host process namespaces and excessive Linux capabilities.
| Configuration | Potential risk |
|---|---|
| privileged: true | Significantly expands container access to host resources |
| hostPID: true | Exposes the host process namespace |
| hostNetwork: true | Places the pod in the host network namespace |
| hostPath | Can expose sensitive host filesystem paths |
| Additional capabilities | May grant powerful kernel-level operations |
| Running as root | Can increase impact when combined with other weaknesses |
Some infrastructure components legitimately require elevated access. Such workloads should be isolated, tightly controlled and reviewed individually.
6.8 — Sandboxed Container Runtimes
Sandboxed runtimes introduce additional isolation mechanisms between workloads and the underlying host.
Examples include technologies that use lightweight virtual machines or user-space kernel interception to reduce exposure to the host kernel.
Standard Containers
Share the host kernel and rely on Linux isolation mechanisms.
MicroVM-Based Isolation
Introduces a virtualisation boundary between workloads and the host.
User-Space Isolation
Uses additional runtime mechanisms to mediate selected application operations.
Sandboxing can improve isolation for selected threat models, but it may introduce compatibility, performance and operational trade-offs.
6.9 — Runtime Threat Detection
Admission controls evaluate workload configuration before execution. Runtime detection examines behaviour after containers begin running.
Runtime monitoring may observe process execution, file access, network activity, privilege changes and other supported operating-system events.
Runtime detection technologies may use eBPF, kernel audit facilities or other supported telemetry mechanisms.
Detection limitation: Runtime monitoring is not automatically prevention. Whether a detected operation can be blocked depends on the monitoring technology, its enforcement capabilities and the configured response.
6.10 — Production Container Security Troubleshooting
| Symptom | Potential cause | Investigation |
|---|---|---|
| Pod rejected at creation | Pod Security Admission violation | Inspect API rejection and namespace policy labels |
| Container fails to start | Security context or runtime incompatibility | Inspect pod events and container logs |
| Permission denied | Filesystem, capability or MAC policy restriction | Inspect runtime permissions and host security events |
| System call rejected | Seccomp profile restriction | Inspect seccomp configuration and relevant logs |
| Unexpected host access | Privileged configuration or exposed host resources | Review pod specification and runtime privileges |
| Suspicious process execution | Potential workload compromise | Correlate runtime events, process context and network activity |
Host-level investigation may require additional operating-system tools and runtime-specific diagnostics.
6.11 — Container Security Engineering Checklist
Section 6 — Engineering Takeaway
Kubernetes container security requires multiple independent controls: Pod Security Admission, restricted security contexts, Linux namespace isolation, capability reduction, seccomp, Mandatory Access Control, runtime hardening and continuous threat detection. Because standard Linux containers share the host kernel, securing the worker node remains essential to protecting the workloads it runs.
Container Runtime Security Explorer
Investigate container isolation, admission security, Linux capabilities, seccomp restrictions, host access and runtime detection. Modify a workload's security posture and observe how different controls affect deployment admission and runtime risk.
Workload security configuration
Container security evaluation ready
Select a workload scenario and evaluate its admission and runtime security posture.
Container security controls
Security layer inspector
Container Isolation Architecture
Kubernetes container security combines admission checks, runtime configuration, Linux isolation mechanisms and operational monitoring.
Select an architecture component to inspect its responsibility.Runtime security console
Container security findings
Kubernetes Secrets, Encryption and Supply Chain Security: KMS, Image Signing, SBOMs and Secure Deployment Pipelines
Kubernetes workloads depend on application images, configuration data, credentials, certificates and external services. Each dependency introduces a potential security risk if its integrity, confidentiality or origin cannot be adequately verified.
An attacker may attempt to compromise a container image, modify a build pipeline, obtain application secrets or deploy an untrusted workload into the cluster.
Production Kubernetes security therefore requires controls across the complete software delivery lifecycle: source integrity, trusted builds, artifact verification, secure registries, admission enforcement, secrets protection and runtime credential management.
7.1 — Secure Kubernetes Software Supply Chain Architecture
A secure software supply chain establishes trust in the components used to build, distribute and execute workloads. The objective is to reduce the likelihood that malicious or unverified software reaches production.
Architecture interpretation: The diagram shows two related but distinct security paths. Software artifacts move through build, verification and deployment. Credentials should be managed through protected secrets systems and delivered only to authorised workloads. Actual verification and secret-delivery mechanisms depend on the selected platform and tooling.
7.2 — Kubernetes Secrets Architecture
Kubernetes Secrets store small amounts of sensitive data such as passwords, tokens and certificates.
A Secret is a Kubernetes API object. Applications may consume Secret data through environment variables, mounted volumes or other supported integration mechanisms.
Secret data should be accessible only to workloads and identities that require it.
Important: Base64 encoding in a Kubernetes Secret manifest is not encryption. Without appropriate encryption at rest and access controls, Secret data may be exposed through API access or underlying storage.
7.3 — Encryption at Rest and etcd Protection
Kubernetes stores cluster state in etcd. This may include Secret objects and other sensitive configuration.
Kubernetes supports encryption at rest for selected API resources when the API server is configured with an encryption provider.
API Access Control
Restricts who can read or modify Secret objects.
Encryption at Rest
Protects selected resource data stored through the API server.
etcd Security
Requires restricted network access, secure transport, backups and operational controls.
Encryption at rest protects stored data but does not replace RBAC, transport encryption or workload-level credential security.
7.4 — Kubernetes KMS Integration
Kubernetes can integrate with an external Key Management Service through a supported KMS provider mechanism.
KMS integration allows encryption operations to depend on externally managed key material, reducing the need to store all encryption secrets directly within the API server configuration.
Engineering consideration: KMS availability, latency, key rotation and recovery procedures can affect Kubernetes API operations. The exact architecture depends on the KMS provider version and implementation.
7.5 — External Secrets Management
Organisations often use external secret-management systems to centralise credential storage, rotation and access control.
Examples include cloud secret managers and dedicated secrets platforms.
Secret Synchronisation
A controller retrieves external secrets and creates or updates Kubernetes Secret objects.
Direct Secret Mounting
Supported drivers or integrations expose external secret material to workloads through mounted volumes.
Security distinction: Synchronising an external secret into a Kubernetes Secret means the data also exists in Kubernetes storage. Direct mounting can reduce that persistence path, but application access, node exposure and credential lifecycle still require protection.
7.6 — Container Image Security
Container images contain application code, operating-system packages, libraries and runtime dependencies.
Vulnerable or malicious components can introduce security risks before a workload starts.
Minimal Images
Reduce unnecessary packages and attack surface.
Vulnerability Scanning
Identify known vulnerabilities in image components.
Trusted Registries
Restrict image sources and protect registry access.
Immutable References
Use image digests to identify specific artifact content.
Image Signing
Verify artifact origin and integrity using trusted signatures.
Build Provenance
Record information about how and where artifacts were produced.
7.7 — Software Bill of Materials (SBOM)
A Software Bill of Materials provides an inventory of software components included in an artifact.
SBOMs can support vulnerability management, dependency analysis, incident response and software supply chain transparency.
Important: An SBOM is an inventory, not proof that an image is secure. Its usefulness depends on completeness, accuracy, provenance and integration with vulnerability and risk management processes.
7.8 — Image Signing and Admission Verification
Image signing provides a mechanism for associating an artifact with a cryptographic signature.
Admission policies can require that deployed images satisfy selected verification conditions before workloads are accepted.
Verification systems must establish which signing identities, keys or attestation authorities are trusted.
Security limitation: A valid signature proves only the properties supported by the verification scheme and trust policy. It does not guarantee that the signed software contains no vulnerabilities or malicious functionality.
7.9 — Secure CI/CD Pipeline Architecture
The CI/CD pipeline is a high-value target because it can produce artifacts that are later executed in production.
Pipeline security should protect source repositories, build workers, credentials, artifact registries and deployment permissions.
| Pipeline stage | Security control |
|---|---|
| Source | Protected branches, code review and access controls |
| Build | Isolated build environments and controlled dependencies |
| Test | Security testing and vulnerability analysis |
| Artifact | Digest pinning, signing and provenance records |
| Registry | Access restrictions and artifact retention policies |
| Deployment | Admission verification and least-privilege deployment identities |
| Runtime | Monitoring, vulnerability response and credential rotation |
7.10 — Production Secrets and Supply Chain Troubleshooting
| Symptom | Potential cause | Investigation |
|---|---|---|
| Pod cannot read Secret | Missing Secret, wrong reference or mount issue | Inspect pod events and Secret references |
| External secret not synchronised | Controller, identity or provider failure | Inspect external secrets controller status |
| ImagePullBackOff | Registry authentication, image reference or network issue | Inspect image pull events and registry access |
| Deployment rejected | Signature or admission policy failure | Inspect admission response and verification logs |
| Secret rotation not reflected | Application reload or delivery mechanism issue | Inspect secret update and workload consumption |
| Unexpected image vulnerability | Outdated dependency or incomplete scanning | Review image digest, SBOM and vulnerability data |
Avoid displaying or exporting Secret values during routine troubleshooting. Use metadata, events and controlled diagnostics where possible.
7.11 — Secrets and Supply Chain Security Checklist
Section 7 — Engineering Takeaway
Kubernetes secrets and supply chain security require coordinated controls across credential storage, encryption, key management, trusted builds, artifact integrity, image registries, admission verification and secure deployment automation. The objective is to ensure that production workloads execute approved software while receiving only the sensitive information required for their operation.
Secure Container Deployment Pipeline
Investigate image provenance, vulnerability scanning, software bills of materials, registry integrity, admission verification and secret protection. Inject supply-chain failures and determine whether a container image should be admitted to a production Kubernetes cluster.
Supply-chain investigation
Secure deployment evaluation ready
Select a supply-chain scenario and inspect its deployment security controls.
Supply-chain security controls
Security component inspector
Software Supply-Chain Security
Secure deployment requires more than a successful image build. Provenance, integrity, vulnerability exposure and secret access are separate controls.
Select a pipeline component to inspect its role.Deployment decision console
Supply-chain engineering findings
Kubernetes Security Observability and Threat Detection: Audit Logs, Hubble, Runtime Events and SIEM Correlation
Kubernetes security controls can restrict access and reduce attack surfaces, but preventive controls alone cannot guarantee that a cluster will remain uncompromised.
Security teams need visibility into API operations, network communication, container execution, identity activity and changes to cluster configuration. These signals help investigators distinguish legitimate operations from potentially malicious behaviour.
A mature Kubernetes security monitoring architecture combines control-plane audit logs, network flow telemetry, runtime security events, identity signals and centralised event correlation.
8.1 — Kubernetes Security Telemetry Architecture
Security telemetry originates from multiple layers of the Kubernetes environment. Each source provides a different view of workload and cluster behaviour.
Architecture interpretation: The diagram represents a logical security monitoring pipeline. Different platforms collect, process and retain telemetry in different ways. Detection quality depends on event coverage, collection reliability, time synchronisation, context enrichment and rule design.
8.2 — Kubernetes API Audit Logging
Kubernetes audit logs record selected API activity according to the configured audit policy.
Audit records can identify the user or service account associated with a request, the requested operation, the affected resource and the resulting response status.
Identity
Which user or service account made the request?
Operation
Which Kubernetes resource or action was requested?
Outcome
Was the operation accepted, rejected or otherwise unsuccessful?
Audit policies determine which events and metadata are captured. Some policies may omit request or response bodies to reduce sensitive data exposure and storage cost.
8.3 — Network Flow Observability with Hubble
Hubble provides network observability for supported Cilium-managed environments.
Flow records can help investigators understand communication between workloads and identify traffic that was permitted or dropped by policy.
Important: A dropped connection does not automatically indicate malicious activity. Misconfiguration, application changes and legitimate troubleshooting can produce similar events. Correlate flow data with workload identity and application context.
8.4 — Container Runtime Security Events
Runtime security monitoring observes selected behaviour of executing containers and host processes.
Depending on the monitoring technology, events may include process execution, file access, privilege changes and suspicious system activity.
| Runtime signal | Potential significance |
|---|---|
| Unexpected shell execution | May indicate interactive access or application exploitation |
| Access to sensitive files | May indicate credential discovery or configuration inspection |
| Privilege-related operation | May indicate attempted privilege escalation |
| Unusual outbound connection | May indicate command-and-control or unauthorised communication |
| Unexpected binary execution | May indicate malware execution or unauthorised tooling |
| Container-to-host interaction | May indicate a risky configuration or attempted boundary crossing |
Runtime events should be evaluated against expected application behaviour and the workload's security configuration.
8.5 — Identity and Authorisation Telemetry
Identity systems provide signals about authentication attempts, token usage, certificate issuance, credential rotation and authorisation decisions.
These signals can help identify unusual service account activity, unexpected identity use or repeated access denials.
Identity telemetry should be correlated with API audit records, workload events and network flows.
8.6 — Security Event Correlation
Individual events often provide incomplete evidence. Correlation combines signals from multiple sources to reconstruct a possible sequence of security-relevant activity.
Correlation rules should consider time windows, workload identity, namespace, pod metadata, node information and other available context.
Engineering consideration: Pod names and IP addresses can change as workloads are rescheduled. Stable application labels, workload ownership metadata and identity context can improve event correlation.
8.7 — Kubernetes Security Incident Timeline
The following scenario illustrates how multiple telemetry sources can contribute to a security investigation.
Investigation principle: This is a hypothetical scenario, not evidence of a real incident. A reliable investigation must validate timestamps, event relationships and alternative explanations before concluding that a workload is compromised.
8.8 — SIEM Integration and Detection Engineering
Security Information and Event Management platforms can centralise Kubernetes telemetry alongside cloud, identity, network and endpoint events.
A detection pipeline typically collects events, normalises fields, enriches context, applies detection logic and generates alerts for investigation.
Collection
Ingest audit, network and runtime telemetry.
Normalisation
Standardise timestamps, identities and event fields.
Enrichment
Attach workload, namespace and asset context.
Detection
Evaluate rules and behavioural signals.
Investigation
Correlate alerts and validate evidence.
Response
Contain threats and preserve relevant evidence.
8.9 — Detection Quality and False Positives
A useful detection system must balance visibility with operational accuracy.
Excessive alert volume can overwhelm analysts, while overly restrictive detection logic may miss important activity.
| Detection challenge | Engineering response |
|---|---|
| High false-positive rate | Improve application baselines and contextual filtering |
| Missing telemetry | Validate collectors, permissions and data pipelines |
| Unreliable timestamps | Synchronise clocks and preserve event timing metadata |
| Changing pod identities | Correlate workload ownership and stable metadata |
| High storage cost | Apply appropriate retention and collection policies |
| Sensitive log content | Restrict access, minimise collection and protect storage |
8.10 — Production Security Observability Troubleshooting
API audit logs are typically collected through control-plane audit configurations or managed-platform integrations. They are not necessarily available through kubectl commands.
8.11 — Kubernetes Security Monitoring Checklist
Section 8 — Engineering Takeaway
Kubernetes security observability requires visibility across API activity, workload communication, runtime execution, identity behaviour and security policy decisions. Correlating these signals helps security teams identify potential compromises, investigate lateral movement and distinguish malicious activity from configuration or operational failures.
Kubernetes Threat Detection Timeline
Investigate suspicious Kubernetes activity across API audit events, container runtime telemetry, network-flow observations and identity signals. Correlate events into an attack timeline and determine whether the available evidence supports a security alert.
Threat detection investigation
Threat investigation ready
Select a scenario and advance through the available security telemetry.
Correlated event timeline
Event inspector
Security Event Correlation
A detection engine combines multiple signals to distinguish suspicious behaviour from ordinary cluster activity.
Select a timeline event to inspect its evidence.Detection console
Threat detection findings
Production Kubernetes Security Troubleshooting and Incident Response: Compromised Pods, Lateral Movement, RBAC Failures and Recovery
Kubernetes security incidents frequently involve multiple infrastructure layers. A suspicious network connection may originate from a compromised application, an incorrectly configured NetworkPolicy, excessive RBAC permissions or a workload using unexpected credentials.
Effective incident response requires engineers to understand how these components interact. Investigators must distinguish configuration failures from malicious behaviour while preserving evidence and minimising disruption to legitimate services.
A production-ready response process combines rapid detection, structured investigation, controlled containment, credential protection, evidence preservation, recovery and post-incident hardening.
9.1 — Kubernetes Security Incident Architecture
Consider a hypothetical incident in which a frontend application is suspected of being compromised. The workload begins making unexpected Kubernetes API requests and attempts to communicate directly with a protected database.
Architecture interpretation: The diagram represents a hypothetical security incident. A blocked connection or rejected API request is not, by itself, proof of compromise. Investigators must correlate runtime activity, workload identity, API audit events, network flows and expected application behaviour.
9.2 — Kubernetes Security Incident Response Lifecycle
A structured response process reduces the risk of destroying evidence, interrupting legitimate workloads or allowing an attacker to continue operating.
Detect
Identify suspicious activity through security telemetry.
Triage
Assess severity, scope and potential business impact.
Investigate
Correlate API, network, runtime and identity evidence.
Contain
Restrict suspicious activity while preserving evidence.
Recover
Restore trusted workloads and validate security.
Improve
Address root causes and strengthen preventive controls.
9.3 — Investigating a Suspected Compromised Pod
The investigation should begin by identifying the affected workload and its operational context.
Relevant information includes the namespace, pod owner, container image digest, service account, worker node, network policies and recent deployment changes.
These commands provide configuration and operational context but do not establish whether a compromise occurred.
Evidence preservation: Deleting or restarting a suspicious pod may destroy volatile evidence. Before taking disruptive actions, follow the incident-response plan and preserve relevant logs, metadata and forensic information where feasible.
9.4 — Investigating Kubernetes RBAC Failures
Unexpected Kubernetes API requests can indicate misconfiguration, automation errors or possible misuse of workload credentials.
The investigation should identify the request identity, resource, verb, namespace and authorisation result.
The impersonation command requires appropriate impersonation permissions. API audit records may provide additional evidence about actual requests.
Engineering distinction: An RBAC denial may show that a security boundary worked as intended. It does not necessarily indicate that the requesting workload was malicious.
9.5 — Investigating NetworkPolicy and Cilium Denials
A blocked network connection may be caused by ingress isolation, egress isolation, incorrect labels, missing allow rules or additional Cilium-specific policies.
Investigators should evaluate both communication endpoints and determine which security controls apply.
| Observation | Potential explanation |
|---|---|
| Ingress denied | Destination workload does not permit the source |
| Egress denied | Source workload does not permit the destination |
| Unexpected allowed traffic | Another additive policy or different enforcement behaviour |
| DNS resolution failure | DNS egress blocked or DNS infrastructure unavailable |
| Service connectivity failure | Policy, service forwarding, endpoint or routing issue |
9.6 — Detecting Lateral Movement
Lateral movement occurs when an attacker attempts to extend access from an initially compromised system to other resources.
In Kubernetes, possible targets include internal services, databases, other workloads, cluster APIs and accessible node resources.
Unusual network activity should be compared with known application dependencies, deployment changes and expected communication patterns.
9.7 — Credential Exposure and Service Account Misuse
A compromised workload may expose credentials available to its processes, including projected service account tokens, application secrets or external service credentials.
The potential impact depends on the permissions associated with those credentials.
Identify Exposure
Determine which credentials were accessible to the affected workload.
Assess Permissions
Review Kubernetes RBAC and external system authorisation.
Rotate Credentials
Revoke or rotate affected credentials through approved procedures.
Important: Deleting a pod does not guarantee immediate invalidation of every credential it accessed. Token lifetimes, external identity systems and secret rotation mechanisms must be considered.
9.8 — Kubernetes Incident Containment
Containment actions should reduce attacker access while preserving necessary evidence and avoiding uncontrolled disruption.
| Containment action | Consideration |
|---|---|
| Restrict network communication | Apply validated policies and confirm effective isolation |
| Revoke credentials | Assess token validity, rotation and external access |
| Restrict API permissions | Review effective RBAC and affected workloads |
| Isolate affected nodes | Consider cluster availability and forensic requirements |
| Stop compromised workloads | Preserve evidence before termination where possible |
| Block malicious artifacts | Prevent redeployment through registry and admission controls |
Containment principle: Containment should be proportionate to the confirmed or suspected threat. High-impact actions require coordination with operational and security stakeholders.
9.9 — Evidence Preservation and Forensic Investigation
Kubernetes workloads are ephemeral. Pods may disappear, containers may restart and nodes may be replaced during normal operations.
Incident response procedures should account for this short-lived infrastructure.
The exact collection procedure depends on platform capabilities, legal requirements and incident-response policy.
9.10 — Recovery and Security Validation
Recovery should restore trusted application functionality while reducing the likelihood of recurrence.
Rebuilding affected workloads from verified artifacts is often preferable to continuing to operate containers whose integrity cannot be established.
Remove Root Cause
Address exploited vulnerabilities or misconfigurations.
Restore Trusted State
Deploy verified images and configuration.
Validate Security
Test RBAC, network policy and runtime controls.
Restore Service
Confirm application health and expected dependencies.
Monitor Closely
Watch for recurring indicators and unexpected behaviour.
Improve Controls
Update detection, hardening and response procedures.
9.11 — Production Kubernetes Security Incident Checklist
Section 9 — Engineering Takeaway
Production Kubernetes security troubleshooting requires a multi-layer investigation across workload configuration, API access, network enforcement, identity, runtime behaviour and security telemetry. The strongest incident-response processes preserve evidence, validate root causes, contain threats carefully and restore workloads from trusted configurations and artifacts.
Kubernetes Security Incident Investigation Lab
Investigate a production Kubernetes security incident. Collect API audit, runtime and network evidence; identify the root cause; select containment measures; and verify whether the cluster has returned to a controlled security posture.
Production incident investigation
Production incident active
Collect evidence before selecting containment and recovery actions.
Evidence collection
Investigation inspector
Incident Response Architecture
Effective incident response requires evidence collection, containment, root-cause analysis, remediation and recovery verification.
Select an architecture component to inspect its operational role.Investigation console
Incident response assessment
- BGP Troubleshooting: Real-World Failures - November 18, 2026
- Kubernetes Security Architecture - November 18, 2026
- AI Networking Architecture: Ethernet, RoCEv2 and InfiniBand - November 18, 2026
