microservices development

Microservices Observability

NETWORK INSIGHT · MICROSERVICES OBSERVABILITY

Monitoring Microservices: Understanding the Distributed System

Modern applications are no longer confined to a single server or a single execution environment. A request may cross API gateways, independently deployed services, container platforms, databases and external dependencies before producing a response.

This distributed architecture introduces flexibility, scalability and independent deployment. However, it also creates a fundamental operational challenge: understanding what is happening when application behaviour becomes unpredictable.

A service may appear healthy while its downstream dependency is experiencing high latency. An application may return successful responses while its performance gradually deteriorates. A single failing dependency can trigger retries, queue growth and cascading delays across multiple services.

The engineering challenge: Knowing that an application is slow is not enough. Engineers need to determine which request was affected, which services participated, where the delay originated, and how the problem propagated through the architecture.

From Application Request to Operational Insight

A conceptual request journey across independently deployed microservices

01
Client Request Application entry point
02
API Gateway Routing and ingress
03
Order Service Application logic
04
Database Persistent dependency
M
Metrics

Request rate, latency, errors and resource saturation.

L
Logs

Detailed events, exceptions, state changes and context.

T
Distributed Traces

Request paths, span relationships and execution timing.

Unified Observability

Correlating telemetry across services helps engineers connect symptoms to dependencies, identify bottlenecks and investigate failures.

01 / DETECT Recognise Degradation

Identify abnormal latency, rising error rates, unusual traffic patterns and resource pressure.

02 / CORRELATE Connect the Signals

Bring together metrics, logs and traces to understand the behaviour of a distributed request.

03 / INVESTIGATE Find the Cause

Follow service dependencies, identify critical paths and isolate likely sources of failure.

Throughout this engineering guide, we will explore how monitoring evolves into observability, how distributed tracing exposes the critical path, how metrics and structured logs provide operational evidence, and how instrumentation, cloud-native monitoring and automated investigation help maintain reliable microservices applications.
Microservices • Observability

Microservices Observability Explorer

Understand how metrics, logs and distributed traces work together to reveal what is happening inside a distributed application.

OBSERVABILITY SIGNALS
Microservices Application Distributed services and dependencies Metrics Rate • Errors • Latency Saturation • Throughput Logs Events • Errors • State Detailed forensic data Distributed Traces Trace ID • Spans End-to-end request path Unified Observability Correlate metrics + logs + traces Investigate the Failure Detect • correlate • explain
LEARNING PANEL
Selected

Unified Observability

Observability becomes powerful when different telemetry signals are correlated rather than examined independently.

QUESTION What is happening?
SIGNAL Metrics + logs + traces
VALUE Connect symptoms to causes
01 MICROSERVICES OBSERVABILITY · SECTION 01

Microservices Monitoring: From Visibility to Understanding

Microservices change how applications are designed, deployed and operated. Instead of running application functionality inside a single tightly integrated system, individual services are distributed across containers, hosts, clusters and cloud environments.

This architecture provides teams with greater flexibility to develop and scale individual services. However, it also introduces new operational complexity. Services communicate over networks, depend on external systems and may be deployed independently by different engineering teams.

A single user transaction may involve authentication, application logic, payment processing, databases, message queues and external APIs. Understanding the health of any individual component does not necessarily explain the behaviour of the complete transaction.

The central challenge is the growing distance between the cause of a failure and its visible effect. A database slowdown may first appear as an API latency problem. A failing downstream service may cause upstream retries, increasing traffic and resource consumption across otherwise healthy services.

01 / Architecture Evolution

From Monolithic Applications to Distributed Services

Traditional monolithic applications generally package multiple functional components into a common deployment unit. While these applications can still be complex and distributed, engineers often have fewer independently deployed services and network boundaries to investigate.

Microservices separate application capabilities into independently deployable components. Each service may have its own runtime, scaling characteristics, logs, metrics and operational dependencies.

Monolithic Architecture Shared Deployment
Application Runtime
Authentication
Business Logic
Data Access
Application Monitoring

Multiple application capabilities operate within a common deployment boundary.

Microservices Architecture Distributed Deployment
API Gateway
Auth Service
Order Service
Data Service

Independently deployed services communicate through APIs and network connections.

The operational difference is significant. In a monolithic application, engineers may investigate application logs, runtime performance and database activity within a relatively consolidated environment.

In a microservices architecture, the same investigation may require correlating telemetry from dozens or hundreds of services, container instances, orchestration components and network connections.

02 / Distributed System Challenges

Why Microservices Are Difficult to Monitor

The characteristics that make microservices attractive also introduce new failure modes. Independent deployments, elastic scaling and distributed communication make the application more dynamic, but they also reduce the usefulness of monitoring components in isolation.

01
Obscured Failure Paths

The source of a failure may be several service calls away from the user-facing symptom. Dependencies can hide the original cause of degradation.

02
Independent Components

Services have different release cycles, scaling behaviour and operational states. One service can fail while others continue operating.

03
Decentralised Execution

Hundreds of service instances may run across multiple hosts and clusters. Requests must be correlated across changing infrastructure.

These challenges become particularly visible during partial failures. An application does not always fail completely. Instead, a small percentage of requests may experience high latency, timeouts or inconsistent results.

Traditional infrastructure checks may report that every server is reachable and that CPU utilization remains within acceptable limits, while customers continue experiencing degraded application performance.

03 / Cascading Failures

How a Single Dependency Can Affect the Entire Application

Microservices are independently deployed, but they are not operationally independent. A request frequently depends on several downstream services completing their work within an acceptable time.

When one dependency slows down, upstream services may wait longer for responses. Retries can increase traffic, connection pools may become exhausted and queues may begin to grow. Without suitable timeouts, backpressure and resilience controls, a local problem can spread across the application.

Example: Cascading Latency Incident
STAGE 01 Database Slowdown Queries exceed their normal response time.
STAGE 02 Service Latency The order service waits for database responses.
STAGE 03 Retry Pressure Timeouts and retries increase application load.
STAGE 04 User Impact API requests become slow or fail.

Engineering observation: The API gateway may report increased request latency, even though the original problem is a database dependency several service boundaries away. This is an illustrative failure scenario, not a live telemetry measurement.

This illustrates why monitoring individual components is insufficient. Engineers need visibility into service relationships, request dependencies, execution timing and the propagation of failures across the distributed system.

04 / Monitoring and Observability

Monitoring Tells Us What Happened. Observability Helps Explain Why.

Monitoring and observability are closely related, but they address different aspects of operating distributed applications.

Monitoring focuses on collecting and evaluating known indicators of system health. Engineers define measurements, dashboards and alerts that identify expected failure conditions or performance deviations.

Observability extends this capability by providing enough contextual telemetry to investigate system behaviour, including failure modes that were not anticipated when monitoring rules were created.

M
Traditional Monitoring
What is happening?

Monitoring tracks known conditions using predefined metrics, health checks, thresholds and alerts.

  • Is the service available?
  • Has CPU utilization increased?
  • Is the error rate above its threshold?
  • Has response time exceeded its objective?
O
System Observability
Why is it happening?

Observability connects telemetry and execution context to investigate unexpected system behaviour.

  • Which dependency caused the delay?
  • Which requests were affected?
  • Where did the failure propagate?
  • What changed before degradation began?

Monitoring remains essential. Alerts, dashboards and service health indicators provide the initial signals that something requires attention. Observability makes those signals more useful by enabling deeper investigation into their causes.

For example, a monitoring system may detect that the 99th-percentile request latency has increased. Distributed tracing can identify which downstream operation consumed the additional time, while structured logs may reveal a database timeout, connection error or unexpected application state.

05 / Visibility Models

White-Box and Black-Box Monitoring

Effective microservices monitoring combines external observations with telemetry collected from inside applications and infrastructure.

B
Black-Box Monitoring

Observes a system from the outside, using synthetic requests, endpoint checks and externally measured availability or latency. It helps determine whether the service behaves correctly from a user's perspective.

W
White-Box Monitoring

Uses internal instrumentation to expose application state, request processing, resource consumption, dependencies and execution details. It helps engineers investigate why the system behaves as it does.

Both approaches are necessary. Black-box monitoring validates the externally observable experience, while white-box monitoring provides internal evidence for diagnosing failures.

A service can pass internal health checks while failing to deliver acceptable end-user performance. Conversely, an external failure may be difficult to diagnose without internal metrics, logs and distributed traces.

06 / Engineering Perspective

Observe the Request, Not Just the Infrastructure

The most important unit of investigation is often the complete user request. Engineers need to understand the sequence of operations that contributed to the response, including service execution, network communication, database queries and external calls.

This requires connecting application-level telemetry with infrastructure and platform signals. Kubernetes pod restarts, network timeouts, service-mesh errors, database saturation and application exceptions may all contribute to the same incident.

A reliable observability strategy therefore extends beyond collecting large volumes of data. It requires meaningful instrumentation, consistent metadata, service dependency context and the ability to correlate signals across the application.

Network Insight · Engineering Principle

A healthy component does not guarantee a healthy application. Microservices reliability must be evaluated across the complete request path. The goal is to connect application symptoms to the services, dependencies and infrastructure conditions responsible for them.

Section 01 — Key Engineering Takeaways
✓ Microservices introduce distributed dependencies, independently deployed components and more complex failure paths.
✓ Local service health does not necessarily represent end-to-end application health.
✓ Monitoring detects known symptoms, while observability provides context for investigating unexpected behaviour.
✓ Black-box and white-box monitoring provide complementary perspectives on system reliability.
✓ Understanding complete request journeys is essential for diagnosing cascading failures and distributed latency.
02
NEXT SECTION Metrics, Logs & Distributed Traces
01 Interactive Engineering Lab

Microservices Architecture Explorer

Investigate a distributed checkout platform. Select services to inspect their health, inject failures, execute requests and observe how dependency problems affect the wider application.

ALL SERVICES HEALTHY
Healthy Degraded Failed

Select any service node to inspect its simulated metrics and dependencies. Animated links show the most recent request path. This is an educational model, not a live Kubernetes or APM connection.

Requests 0
Success Rate —
Last Latency —
Distributed Request Trace No request yet
Select a scenario and press Send Request to generate a trace.
✓
Architecture Ready

Start with the healthy scenario, then inject a dependency failure to investigate how microservices propagate errors.

Engineering principle: A service can be locally healthy while an end-to-end transaction fails because one of its dependencies is unavailable or slow. This simplified simulation assumes sequential payment and inventory calls, with no retries, caching or fallback behaviour.
02 MICROSERVICES OBSERVABILITY · SECTION 02

Metrics, Logs & Distributed Traces

Understanding a distributed application requires more than knowing whether its individual services are running. Engineers must observe performance trends, investigate individual events and reconstruct how requests move between services.

Three complementary telemetry signals form the foundation of modern microservices observability: metrics, logs and distributed traces. Each provides a different perspective on application behaviour, and each becomes more valuable when correlated with the others.

Metrics describe measurable system behaviour over time. Logs record individual events and application context. Distributed traces reconstruct operations across service boundaries, helping engineers understand request execution and dependency timing.

The objective is not simply to collect three independent datasets. It is to connect them so that an abnormal metric can lead to a specific request trace, which can then lead to the relevant application logs.

01 / The Three Observability Signals

Three Perspectives on the Same Distributed System

A microservices application produces different types of operational information. These signals answer different engineering questions and are suited to different forms of analysis.

M
Metrics
How is the system performing?

Aggregated numerical measurements reveal changes in traffic, latency, errors, resource utilization and saturation.

Example: p99 latency = 850 ms
L
Logs
What events occurred?

Structured event records capture errors, exceptions, state transitions and contextual application information.

Example: database connection timeout
T
Traces
Where did the request spend time?

Distributed spans reconstruct the request journey and expose timing, service dependencies and failures.

Example: Order Service → Database
◎
Correlated Observability

Shared service metadata, trace context and consistent instrumentation connect telemetry into a coherent investigation.

None of these signals is a complete substitute for the others. Metrics provide efficient aggregated visibility but often lack individual request detail. Logs can expose precise events but may not show the complete request path. Traces provide execution context but can be affected by sampling and instrumentation gaps.

02 / Metrics

Metrics: Quantifying Application Health

Metrics are numerical measurements recorded over time. They allow engineers to observe how system behaviour changes, compare current performance against a baseline and identify conditions that require investigation.

Common microservices metrics include request throughput, response latency, error rates, resource utilization, queue depth and connection-pool saturation.

Understanding Time-Series Metrics

A metric measurement generally combines a value with a timestamp and identifying attributes or labels.

For example, request latency can be analysed by service, endpoint, deployment or region, provided the metric instrumentation includes the appropriate dimensions.

  • Request rate measures incoming demand.
  • Error rate identifies failed operations.
  • Latency percentiles reveal slow requests.
  • Saturation indicates resource pressure.
Illustrative Service Metrics
Request throughput 1,250 req/s
p50 latency 42 ms
p99 latency 850 ms
HTTP 5xx error rate 3.2%
CPU utilization 68%

Percentile latency measurements are particularly important. An average response time may appear acceptable even when a subset of users experiences severe delays. The 99th-percentile latency, commonly called p99, describes a threshold at or below which approximately 99% of measured requests fall within the selected measurement window.

Tools such as Prometheus collect and store time-series metrics, while Grafana and other visualization platforms help engineers build dashboards and investigate performance trends.

Metrics should also be designed with storage efficiency in mind. Adding labels creates additional time series, and high-cardinality dimensions such as unique user identifiers or individual request IDs can significantly increase collection and storage costs.

03 / Logs

Logs: Detailed Evidence of Application Behaviour

Logs record individual events generated by applications, infrastructure components and supporting platforms. They provide contextual evidence about errors, authentication activity, configuration changes, service lifecycle events and unexpected application states.

In distributed environments, unstructured text messages are difficult to correlate across independently deployed services. Structured logging improves searchability by representing events using consistent fields and formats.

Structured Logging

A structured log record can contain a timestamp, severity, service name, operation, trace identifier and error details.

Consistent fields make it possible to search across services and connect events belonging to the same distributed request.

  • Timestamp identifies when an event occurred.
  • Service name identifies its source.
  • Trace ID links events to a request.
  • Severity distinguishes event importance.
  • Error context supports investigation.
Example JSON Application Log
{ "timestamp": "2026-11-06T14:32:10Z", "service": "order-service", "severity": "ERROR", "trace_id": "4bf92f...", "operation": "create-order", "error": "database timeout" }

Centralized logging platforms collect events from multiple sources so engineers can search, filter and analyse them without accessing individual service instances.

Technologies such as Elasticsearch, Logstash, Fluent Bit, Fluentd, Grafana Loki and cloud-native logging services support different stages of log collection, processing, storage and analysis.

Effective logging also requires operational discipline. Sensitive data should not be unnecessarily recorded, retention policies should reflect business and security requirements, and log volumes must be controlled to avoid excessive cost and noise.

04 / Distributed Traces

Distributed Traces: Reconstructing the Request Journey

Distributed tracing follows a request as it crosses service boundaries. Instead of viewing isolated application events, engineers can reconstruct the operations that contributed to a complete transaction.

A trace contains spans representing individual units of work. Each span can record the operation name, start time, duration, attributes, status and relationship to its parent span.

Trace context propagates between services, allowing spans generated by different components to be associated with the same request. Standards such as W3C Trace Context support this propagation using identifiers carried in headers such as traceparent.

Understanding a Distributed Trace

A request may pass through an API gateway, authentication service, order service and database.

Each instrumented operation contributes a span to the overall trace, revealing the timing and relationships between different parts of the transaction.

  • Trace ID identifies the request journey.
  • Span ID identifies an individual operation.
  • Parent relationships describe execution structure.
  • Duration highlights expensive operations.
Illustrative Span Durations
API Gateway
24 ms
Auth Service
15 ms
Order Service
210 ms
Database
280 ms

Bars compare individual span durations. They are not a chronological waterfall or a complete end-to-end latency calculation.

Tracing is particularly valuable when failures are distributed across several components. A slow API response may originate from a downstream database, an external service or a network connection rather than the API gateway itself.

Trace visualization systems such as Jaeger, Zipkin and Grafana Tempo help engineers explore service dependencies, execution timing and request-level failures.

05 / Signal Comparison

Choosing the Right Signal for the Investigation

Metrics, logs and traces are complementary. The appropriate starting point depends on the operational question being investigated.

Capability Metrics Logs Traces
Primary purpose Measure trends Record events Reconstruct operations
Typical data Numeric time series Event records Spans and relationships
Best for Health and alerting Error investigation Latency and dependencies
Common context Service labels Trace IDs and metadata Trace and span IDs
Limitation Limited request detail Fragmented without correlation Sampling and missing spans
06 / Correlated Investigation

Connecting the Signals During a Production Incident

Consider an e-commerce application where customers report slow checkout operations. A dashboard shows increasing latency, but the initial metric does not reveal which component is responsible.

Example Incident: Checkout Latency Spike
01
Metrics Detect the Symptom

The checkout endpoint's p99 latency rises from its normal baseline to 850 ms. Error rates also begin increasing.

02
Traces Locate the Delay

Affected request traces show excessive time spent waiting for a downstream database operation.

03
Logs Provide Evidence

Correlated application logs report database connection timeouts, supporting the suspected dependency issue.

Investigation outcome: The telemetry points toward a database dependency problem rather than an isolated API gateway issue. Engineers can now examine database health, connection pools, query performance and network connectivity to confirm the root cause.

This workflow demonstrates the value of correlation. Metrics identify the degradation, traces narrow the affected execution path, and logs provide contextual evidence.

However, telemetry correlation does not automatically prove causation. Engineers should validate their hypothesis against dependency health, configuration changes, infrastructure events and other available operational evidence.

Network Insight · Engineering Principle

Collect telemetry with an investigation question in mind. Metrics should reveal meaningful service health, logs should preserve useful context, and traces should reconstruct critical request paths. Correlation across these signals transforms isolated measurements into operational understanding.

Section 02 — Key Engineering Takeaways
✓ Metrics quantify system behaviour and provide efficient monitoring and alerting.
✓ Structured logs capture detailed events and support cross-service investigation.
✓ Distributed traces reconstruct request execution and reveal service dependencies.
✓ Shared identifiers and consistent metadata enable telemetry correlation.
✓ Effective observability combines all three signals while managing sampling, cardinality, retention and data volume.
03
NEXT SECTION Distributed Tracing & Critical Paths
02 Interactive Engineering Lab

Three Pillars of Observability

Investigate a distributed checkout application using three complementary telemetry signals. Select an incident, inspect metrics, search structured logs, follow distributed traces and identify the most likely fault.

BASELINE HEALTHY
01 / Select a Production Scenario

Metrics — What Is Happening?

PILLAR 01
p99 Latency 305 ms Checkout requests
Error Rate 0% Failed transactions
Throughput 120 Requests / second
High Low Earlier Now

The chart is illustrative. The dashed line represents a demonstration latency threshold, not a universal SLO.

Service Dependency Map

TOPOLOGY
API Gateway
ORD Order
PAY Payment
INV Inventory
Healthy baseline: All four services are available. Checkout requests complete normally.

Metrics Investigation

PILLAR 01

Service-level metrics show normal latency, throughput and error rates.

Engineering question: Are customers experiencing a measurable degradation in application behaviour?
02 / Engineering Investigation Challenge

Identify the Most Likely Incident Cause

Examine the telemetry before choosing an answer. Different scenarios produce different evidence.

Select a hypothesis, then validate it against the available evidence.
Engineering principle: Metrics reveal the extent of a problem, logs provide event-level context and distributed traces show where time is spent across service boundaries. Correlating these signals is more useful than investigating any one in isolation. All values and incident outcomes in this lab are simulated examples.
03 MICROSERVICES OBSERVABILITY · SECTION 03

Distributed Tracing: Following the Critical Path

In a distributed application, a single user request can trigger operations across multiple services, databases and external dependencies. Distributed tracing makes this execution journey visible by connecting individual operations into a coherent end-to-end trace.

Traditional logs and metrics provide valuable operational evidence, but they do not always reveal the sequence of service calls responsible for a particular request. Engineers may know that an API is slow without understanding which downstream operation caused the delay.

Distributed tracing addresses this challenge by recording the relationships and timing between operations. It enables engineers to examine a request across service boundaries, identify expensive dependencies and investigate where failures originated.

The fundamental objective is to reconstruct the complete request journey, rather than investigating each service in isolation.

01 / End-to-End Request Visibility

Following a Request Across Microservices

Consider an application where a customer submits an order. The request enters through an API gateway, passes through authentication, reaches the order service and eventually accesses a database.

Each component performs a different operation. Some calls may execute sequentially, while others can run concurrently. Distributed tracing records these operations as spans and connects them using shared trace context.

Distributed Request Journey Example: GET /orders/1234
01
Client HTTP request
02
API Gateway Ingress and routing
03
Auth Service Identity validation
04
Order Service Business logic
05
Database Data retrieval
Shared Trace Identifier 4bf92f3577b34da6a3ce929d0e0e4736

The trace identifier allows telemetry generated across the participating services to be associated with the same distributed request. Individual operations receive their own span identifiers, while parent relationships describe how the operations are connected.

This distinction is important. A trace represents the broader request execution, while a span represents a particular operation within that execution.

02 / Trace and Span Fundamentals

Understanding Traces, Spans and Context

Distributed tracing relies on a small set of core concepts. Together, these concepts allow operations performed by different services to be reconstructed into a coherent execution model.

T
Trace

A trace represents a distributed operation, typically associated with a request or transaction. It groups related spans under a shared trace ID.

S
Span

A span represents an individual unit of work. It can contain timing, operation names, attributes, events and status information.

C
Trace Context

Trace context carries identifiers and sampling information across process boundaries, allowing distributed spans to remain linked.

A span typically records when an operation began, when it ended and how it relates to other operations. Additional attributes may identify the service, endpoint, database operation or relevant execution environment.

Parent-child relationships form a hierarchy. A root span may represent the incoming API request, while child spans represent authentication, application processing and downstream database calls.

Example Span Hierarchy
Root: API Request Trace root
API Gateway Child span
Order Service Child span
Database Query Child span

In real applications, the span hierarchy may contain branches representing concurrent operations. The structure therefore describes execution relationships rather than simply listing services in the order they appear on a network diagram.

03 / Trace Propagation

How Trace Context Crosses Service Boundaries

A distributed trace is useful only when participating services can preserve and propagate the request context.

When a service receives an instrumented request, it can extract the incoming trace context, create a new span and propagate the relevant context to downstream services.

The W3C Trace Context standard defines interoperable HTTP headers, including traceparent and tracestate, to support trace propagation between different frameworks and platforms.

# Example W3C Trace Context header traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 # Components Version: 00 Trace ID: 4bf92f3577b34da6a3ce929d0e0e4736 Parent ID: 00f067aa0ba902b7 Flags: 01

In this example, the trace ID identifies the distributed trace. The parent ID identifies the span associated with the propagated context, and the flags include the sampled bit.

As the request crosses service boundaries, the trace ID normally remains consistent, while each newly created span receives its own span ID. Outgoing context is updated to reference the appropriate span for the next downstream operation.

Context propagation can be disrupted by missing instrumentation, unsupported messaging integrations or application code that fails to forward the context. These gaps can result in disconnected traces or incomplete request visibility.

04 / Trace Waterfall Analysis

Reading a Distributed Trace Waterfall

A trace waterfall visualizes spans along a common time axis. Each horizontal bar represents an operation, showing when it started and how long it took.

This representation allows engineers to compare execution timing, identify overlapping operations and locate spans associated with excessive latency.

Example Trace Waterfall End-to-end: 400 ms
API Request
400 ms
API Gateway
32 ms
Auth Service
40 ms
Order Service
288 ms
Database
240 ms
0 ms 100 200 300 400 ms

Engineering observation: The database span consumes 240 ms of the 400 ms request window and accounts for much of the order service's execution time. This is illustrative trace data.

A waterfall must be interpreted carefully. Nested spans overlap with their parents, so simply adding every span duration together does not produce the total request latency.

Engineers must also distinguish between elapsed time and exclusive execution time. A service span may include time spent waiting for a downstream dependency rather than performing local computation.

05 / Critical Path Investigation

Finding the Operations That Determine Request Latency

The critical path is the chain of dependent operations that determines the minimum completion time of a request, given the observed execution dependencies and durations.

In sequential request flows, a slow downstream operation can directly increase total response time. In concurrent execution, the longest dependent branch may determine when the request can complete.

01
Identify the Root Span

Establish the total request duration and the start of the transaction.

02
Inspect Child Spans

Compare operation timing and parent-child relationships.

03
Locate the Delay

Identify the dependent operation contributing most to completion time.

04
Validate the Cause

Correlate the trace with logs, metrics and dependency health.

The longest individual span is not automatically the critical path. Engineers must account for execution relationships, parallel work and the timing of dependent operations.

For example, two database queries may execute concurrently. Improving the faster query may have no effect on overall request latency if the slower query still determines when the application can return its response.

06 / Distributed Tracing in Practice

Instrumentation, Collection and Trace Analysis

Distributed tracing requires services to generate spans and propagate context. Instrumentation can be implemented manually within application code or automatically through supported libraries and frameworks.

OpenTelemetry provides vendor-neutral APIs, SDKs, instrumentation libraries and collection components for generating and exporting telemetry.

Traces are commonly exported to a collector or observability backend, where they can be processed, stored and visualized.

Technologies such as Jaeger, Zipkin and Grafana Tempo provide different approaches to storing and exploring distributed traces.

Tracing also introduces operational considerations. Sampling strategies influence which requests are retained, instrumentation overhead must be controlled, and sensitive information should not be unnecessarily included in span attributes or events.

07 / Engineering Case Study

Investigating a Slow Checkout Transaction

Consider an e-commerce application where customers experience intermittent delays during checkout. Infrastructure dashboards show no obvious host failure, but application latency has increased.

Distributed Trace Investigation
STEP 01 / SYMPTOM Checkout Latency

The request takes 1.2 seconds to complete, significantly exceeding its normal baseline.

STEP 02 / TRACE Database Wait Time

The distributed trace reveals that the order service spends most of its time waiting for a database operation.

STEP 03 / EVIDENCE Connection Pressure

Application logs and database metrics indicate connection-pool exhaustion and increased query wait time.

Investigation outcome: The combined telemetry supports a database connection-pool bottleneck as the likely cause of the checkout latency. Engineers should validate connection usage, query behaviour and pool configuration before implementing corrective changes.

This case illustrates why tracing is particularly valuable in distributed systems. The visible symptom appears at the user-facing application, while the likely bottleneck exists several service boundaries downstream.

Distributed tracing narrows the investigation by exposing the execution path, allowing engineers to focus on the dependencies most relevant to the incident.

Network Insight · Engineering Principle

Investigate latency through execution relationships, not isolated service timings. A distributed trace reveals how operations depend on one another, where time is consumed and which downstream calls influence the end-to-end request. Correlating traces with metrics and logs helps turn a suspected bottleneck into a validated diagnosis.

Section 03 — Key Engineering Takeaways
✓ Distributed tracing reconstructs requests across service boundaries.
✓ Traces contain spans linked through identifiers and parent relationships.
✓ W3C Trace Context supports interoperable propagation of distributed tracing context.
✓ Trace waterfalls reveal timing, concurrency and dependency delays.
✓ Critical-path analysis identifies the dependent operations that determine request completion time.
✓ OpenTelemetry and trace backends provide instrumentation, collection and analysis capabilities.
04
NEXT SECTION Metrics, Golden Signals & Cardinality
03 Interactive Engineering Lab

Distributed Trace Investigation Lab

Follow a checkout transaction across microservices. Inspect individual spans, replay request propagation, investigate dependency latency and identify the operations responsible for end-to-end response time.

TRACE HEALTHY
01 / Select a Trace Scenario
02 / End-to-End Trace Metrics
Total Duration 305 ms
Trace Spans 5
Slowest Operation Payment
Trace Outcome Success

Request Propagation Topology

SERVICE GRAPH
USR Client
API Gateway
ORD Order
PAY Payment
INV Inventory

The checkout workflow performs payment before inventory. Each link represents a synchronous request dependency.

Span Inspector

SPAN DETAILS
Payment Authorisation
OK

Payment service authorises the checkout transaction.

Start Offset 100 ms
Duration 110 ms
Span ID SPAN-04
Parent Span Order
Select a span in the waterfall to inspect its timing and parent-child relationship.
03 / Distributed Trace Waterfall

Request Execution Timeline

TRACE-3001
0 ms 25% 50% 75% 305 ms
Root / Service Span Child Span High Latency Failure

Critical Path Analysis

Payment is the slowest individual operation in the healthy checkout request.

Critical-path analysis identifies operations that determine end-to-end completion time.

Trace Context Propagation

A distributed trace connects operations using trace IDs and parent-child span relationships.

Trace ID TRACE-3001
Propagation W3C Context

This lab uses shortened demonstration identifiers. Production W3C trace IDs and span IDs use fixed-length hexadecimal values.

04 / Engineering Investigation

Which Dependency Is Responsible?

Select the operation most responsible for the scenario's abnormal latency or failure. For the healthy baseline, select No Incident.

Investigation Event Timeline

EVENTS
Lab initialised. Healthy trace loaded.
Engineering principle: Distributed tracing reconstructs the execution of a request across service boundaries. Parent spans can overlap child spans, so their durations must not be added together. The critical path is the sequence of dependent operations that determines the end-to-end completion time. This lab models a simplified sequential checkout workflow.
04 MICROSERVICES OBSERVABILITY · SECTION 04

Metrics, Golden Signals & Cardinality

Metrics provide the quantitative foundation of microservices monitoring. They transform application behaviour into measurable time-series data, allowing engineers to identify degradation, establish performance baselines and evaluate whether services are meeting their operational objectives.

In a distributed environment, however, collecting every available measurement does not automatically improve visibility. Engineers must choose meaningful signals, understand how metrics are aggregated, and manage the storage and query costs associated with metric labels.

Effective metrics engineering connects application health with the user experience. Instead of focusing exclusively on CPU, memory and infrastructure availability, teams should measure request latency, traffic, errors and resource saturation across the services that deliver application functionality.

01 / The Four Golden Signals

Establishing a Meaningful Monitoring Baseline

The four golden signals provide a practical starting point for monitoring user-facing services. They focus attention on the measurements most likely to reveal performance degradation and operational pressure.

L
Latency

The time required to serve a request, including the behaviour of slow and unsuccessful operations.

T
Traffic

The amount of demand placed on the service, such as requests per second or transactions.

E
Errors

Failed requests, incorrect responses, timeouts and application-level failures.

S
Saturation

The degree to which a resource approaches its practical capacity or operating limit.

These signals should be measured at meaningful service boundaries. For example, an API gateway can expose overall request traffic and response latency, while a database service can expose connection utilization, query duration and queue pressure.

A useful monitoring baseline captures normal behaviour across different traffic periods. Without a baseline, engineers may struggle to distinguish a genuine incident from expected variations in application demand.

Example: Microservices Health Dashboard Illustrative Telemetry
Request Rate 1,250 Requests per second
p99 Latency 850 ms Elevated response time
Error Rate 3.2% HTTP 5xx responses
Availability 99.92% Example measurement window
Request Latency Trend Illustrative relative values
Earlier Time → Later
Typical latency trend Elevated tail latency

In this example, the elevated p99 latency and error rate suggest that a subset of requests is experiencing degradation. The next step is to correlate the metrics with request traces and application logs to identify the affected services.

02 / Monitoring Methodologies

RED and USE: Different Perspectives on System Health

The golden signals provide a broad monitoring framework. Two complementary methodologies help engineers organise metrics around service behaviour and resource utilization.

RED
Request-Oriented Monitoring

RED focuses on the behaviour of services that process requests. It is particularly useful for APIs, microservices and other transaction-oriented components.

  • Rate: Requests processed per second.
  • Errors: Failed requests or operations.
  • Duration: Time required to process requests.
USE
Resource-Oriented Monitoring

USE focuses on the operational condition of resources and infrastructure components. It helps identify resource bottlenecks and capacity pressure.

  • Utilization: Resource usage relative to capacity.
  • Saturation: Queuing or excess demand.
  • Errors: Resource-level failures.

RED is useful for understanding the customer-facing service experience, while USE helps investigate the infrastructure resources supporting those services.

For example, RED metrics may reveal increasing API response duration, while USE metrics show that a database connection pool or storage device is approaching saturation.

03 / Latency Percentiles

Why Average Latency Can Hide Performance Problems

Average response time provides a useful summary, but it can hide severe delays affecting a smaller percentage of requests.

Percentiles describe the distribution of observed response times. The p50 value represents the median, while p95 and p99 highlight the slower portion of the request population.

Example: Request Latency Distribution
p50
42 ms
p95
260 ms
p99
850 ms

Bars are scaled relative to the largest displayed value. The figures are illustrative rather than live measurements.

A service with a p50 latency of 42 ms may appear responsive for most users, yet a p99 latency of 850 ms indicates that the slowest portion of requests experiences substantially higher delays.

Tail latency is especially important in distributed systems because a transaction may depend on several downstream operations. Slow dependencies can increase end-to-end response time, particularly when requests must wait for multiple services to complete.

04 / Service Objectives

SLIs, SLOs and Error Budgets

Monitoring becomes more useful when measurements are connected to clearly defined service objectives. Instead of alerting on arbitrary thresholds, engineering teams can evaluate whether a service is delivering an acceptable user experience.

A Service Level Indicator (SLI) is a quantitative measurement of service behaviour. A Service Level Objective (SLO) defines the desired performance of that indicator over a specified period.

Service Level Indicator
99.95%

Example objective for the proportion of eligible requests completed successfully during a defined measurement window.

The indicator must define which requests count as eligible and which responses qualify as good.

Error Budget
0.05%

A 99.95% success SLO permits 0.05% of eligible requests to fall outside the objective.

Illustration: 40% of the error budget used

Error budgets help teams balance reliability and the pace of change. When a service consumes its budget rapidly, engineers may prioritise stability improvements over additional deployments.

For a request-based SLO, the error budget can be calculated from the total number of eligible requests during the measurement window. Time-based availability objectives use a different measurement model.

05 / Prometheus and Metric Collection

Collecting and Querying Time-Series Data

Prometheus is a widely used open-source monitoring system designed for collecting, storing and querying time-series metrics. It commonly uses a pull-based model in which the Prometheus server scrapes metrics endpoints.

Applications can expose metrics directly through client libraries, while exporters make measurements available from systems that do not expose Prometheus-compatible metrics natively.

Prometheus supports several metric types, including counters, gauges, histograms and summaries. Each type serves different measurement and aggregation requirements.

# Example Prometheus counter http_requests_total{ service="order-service", method="GET", status="200" } 125000 # Example request rate query (PromQL) sum(rate(http_requests_total[5m])) # Example HTTP 5xx request rate sum(rate(http_requests_total{status=~"5.."}[5m]))

A counter increases as events occur, making it suitable for request totals and error counts. The PromQL rate() function estimates the average per-second increase over a selected range.

Histograms can capture latency distributions and support percentile estimation across aggregated observations. Metric design should reflect the operational questions engineers need to answer.

06 / Metric Labels and Cardinality

Why Metric Cardinality Matters at Scale

Labels add dimensions to metrics. They allow engineers to analyse measurements by service, endpoint, region, deployment or other relevant attributes.

However, every unique combination of metric name and label values creates a distinct time series. As label combinations grow, the number of stored series can increase dramatically.

Example: Cardinality Growth

Consider a single metric with three independently varying label dimensions. The following values illustrate the potential number of unique series.

Service Labels 20 Different service values
Endpoint Labels 50 Different endpoint values
Instance Labels 100 Different instance values
20 services × 50 endpoints × 100 instances = 100,000 potential time series

This is an upper-bound illustration assuming every label combination exists. Actual cardinality depends on which combinations are emitted.

High-cardinality labels can increase memory consumption, storage requirements, query complexity and telemetry costs. This becomes particularly problematic when labels contain values that change for every request.

Unique request identifiers, timestamps and unrestricted user identifiers are generally poor choices for Prometheus metric labels because they can create an extremely large number of unique time series.

Such identifiers are usually better suited to structured logs or distributed traces, where they provide request-level correlation without creating a new metric series for every transaction.

07 / Aggregation and Alert Engineering

Turning Metrics into Actionable Alerts

Metrics are commonly evaluated over aggregation windows. The selected window affects how quickly a monitoring system detects changes and how much short-term variation appears in the resulting measurements.

Short windows can reveal rapid degradation but may produce noisy alerts. Longer windows smooth temporary fluctuations but can delay detection of emerging problems.

Effective alerting balances these characteristics. Alerts should represent meaningful operational conditions rather than every minor deviation from normal metric behaviour.

SLO-based alerting can use error-budget burn rates to identify conditions that threaten reliability objectives. Multi-window alerting strategies can combine rapid detection with protection against transient metric noise.

Monitoring systems should also account for missing telemetry, service discovery changes, delayed collection and differences between scrape intervals and query evaluation windows.

08 / Engineering Case Study

Investigating Elevated Tail Latency

Consider an order-processing service that normally handles 1,250 requests per second. Its average latency remains relatively stable, but p99 latency begins increasing during periods of peak demand.

Production Monitoring Investigation
STEP 01 / DETECT Latency Degradation

p99 latency rises to 850 ms while request traffic increases above its normal operating baseline.

STEP 02 / CORRELATE Resource Saturation

Database connection-pool utilization approaches capacity, and query wait times increase.

STEP 03 / INVESTIGATE Dependency Bottleneck

Distributed traces show longer waits on database operations, supporting a connection-capacity bottleneck hypothesis.

Engineering outcome: The combined RED and USE signals identify a likely database resource bottleneck. Engineers can validate connection-pool configuration, query performance and workload behaviour before applying corrective changes.

This investigation demonstrates why monitoring should connect user-facing performance with resource-level measurements. Request latency reveals the impact, while saturation metrics help explain the underlying pressure.

Network Insight · Engineering Principle

Measure what affects the user, then correlate it with what constrains the system. Golden signals and RED metrics reveal service degradation, while USE metrics expose resource pressure. SLIs and SLOs establish reliability expectations, and careful label design keeps the monitoring system scalable.

Section 04 — Key Engineering Takeaways
✓ The four golden signals are latency, traffic, errors and saturation.
✓ RED measures request behaviour, while USE examines resource-level conditions.
✓ Percentile latency measurements reveal slow requests that averages may conceal.
✓ SLIs, SLOs and error budgets connect telemetry to reliability objectives.
✓ Prometheus provides metric collection, storage and time-series querying.
✓ High-cardinality labels increase monitoring costs and should be controlled through deliberate metric design.
05
NEXT SECTION Centralised Logging & Log Analysis
04 Interactive Engineering Lab

Live Metrics & SLO Dashboard

Explore the four golden signals, RED metrics, latency percentiles and service-level objectives. Inject production incidents, inspect changing telemetry and investigate how service degradation consumes an error budget.

MONITORING HEALTHY
01 / Monitoring Configuration
02 / Four Golden Signals
Latency — p99 305 ms

Tail response time

Traffic — Rate 120

Requests per second

Errors — Rate 0.10%

Unsuccessful requests

Saturation — Utilisation 38%

Illustrative capacity usage

Live Latency Telemetry

ROLLING SAMPLES
High Low Earlier Now
Observed p99 latency Illustrative threshold

Samples update while traffic is running. The chart represents simulated p99 latency rather than measurements from a real monitoring system.

Latency Distribution

PERCENTILES
p50 85 ms
p95 180 ms
p99 305 ms
p50 — Median 85 ms
p95 — Tail 180 ms
p99 — Extreme Tail 305 ms
Engineering insight: Median latency can remain acceptable while a small proportion of requests experience severe delays. Tail percentiles expose those slow requests.
03 / Service Level Objectives & Error Budget

SLO Control Panel

RELIABILITY ENGINEERING
Allowed Failures 0.10%
Observed Error Rate 0.10%
Burn Rate 1.0×
Error Budget Remaining

Calculating error budget...

SLOs establish a reliability objective. Error budgets quantify the acceptable proportion of unsuccessful requests.

RED Metrics Explorer

SERVICE HEALTH
Rate — Requests / second 120
Errors — Failed requests 0.10%
Duration — p99 latency 305 ms
RED metrics summarise request rate, error rate and duration. They help distinguish increased load from degraded service behaviour.
04 / Incident Investigation Challenge

What Is the Primary Operational Problem?

Select a traffic scenario, examine the four golden signals and choose the diagnosis that best matches the simulated telemetry.

Investigate the telemetry and select the most likely operational condition.

Monitoring Event Timeline

EVENTS
Engineering principle: The four golden signals describe latency, traffic, errors and saturation. RED metrics provide a request-focused view of service behaviour. SLOs translate reliability goals into measurable objectives, while error budgets help teams balance service reliability and delivery velocity. All telemetry is simulated. The budget display uses an illustrative constant-rate projection rather than historical production accounting.
05 MICROSERVICES OBSERVABILITY · SECTION 05

Centralised Logging: From Events to Evidence

Logs provide detailed evidence of application behaviour. They record errors, state changes, requests, warnings and operational events that help engineers understand what happened inside individual services.

In a monolithic application, engineers may be able to investigate a problem using a relatively small collection of application logs. In a microservices environment, however, relevant events can be distributed across many services, containers, hosts and clusters.

A single failed transaction may generate log events in an API gateway, authentication service, order service, database client and external dependency. Without centralised collection and consistent context, connecting those events becomes difficult.

Centralised logging transforms isolated application messages into searchable, correlated operational evidence. It enables engineers to investigate incidents across the complete distributed system rather than examining individual components independently.

01 / Distributed Logging Challenges

Why Traditional Log Investigation Breaks Down

Microservices generate large volumes of operational events. These events may be produced by short-lived containers, independently deployed services and infrastructure components that scale dynamically with application demand.

Logs stored only on individual hosts or container filesystems can become difficult to access, especially when workloads are rescheduled or terminated.

01
Distributed Sources

Relevant events are spread across different services, containers, hosts and cloud environments. Engineers need a unified view.

02
Ephemeral Infrastructure

Containers and pods may restart or disappear. Local log files cannot always be relied upon for historical investigation.

03
Volume and Noise

High request rates can produce enormous log volumes. Poorly designed logging increases cost and obscures useful evidence.

Centralised logging addresses these challenges by collecting events from multiple sources, applying consistent processing and making the resulting records searchable through a common investigation interface.

02 / Centralised Logging Architecture

How Logs Move Through a Collection Pipeline

A typical logging architecture contains application log producers, collection agents, processing components, storage systems and analysis interfaces.

Applications emit log events to standard output, files or supported logging interfaces. Collectors forward the events to a processing or storage platform, where they can be indexed, labelled, retained and queried.

Distributed Logging Pipeline

From microservices to searchable operational evidence

01
Microservices Application events and exceptions
02
Collectors Fluent Bit, Beats or agents
03
Processing Parsing, enrichment and filtering
04
Log Storage Searchable event retention
05
Analysis Search, dashboards and investigation

Architecture note: This is a conceptual pipeline. Some platforms combine processing, storage and analysis, while others use separate services or buffering systems between collection stages.

In Kubernetes environments, logging agents may run as DaemonSets to collect container logs from cluster nodes. Other architectures use sidecars, application-level exporters or managed cloud collection services.

Reliable collection pipelines should also consider buffering, backpressure, delivery failures and retention policies. If a central logging destination becomes unavailable, an unbuffered pipeline may lose events during the outage.

03 / Structured Logging

Designing Logs for Search and Correlation

Free-form text logs can be useful for human inspection, but they are harder to query consistently across multiple independently developed services.

Structured logging represents events using defined fields, commonly encoded as JSON. This enables filtering by service, severity, operation, trace ID, deployment and other contextual attributes.

Recommended Log Context

Useful log fields depend on the application, but a consistent schema improves the ability to investigate distributed failures.

  • Timestamp: When the event occurred.
  • Severity: DEBUG, INFO, WARN or ERROR.
  • Service: Which component generated the event.
  • Trace ID: Which distributed request it belongs to.
  • Operation: The action being performed.
  • Error details: The failure or exceptional condition.
Example Structured JSON Log
{ "timestamp": "2026-11-06T14:32:10Z", "severity": "ERROR", "service": "order-service", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "operation": "create-order", "error": "database timeout" }

Structured logging is especially valuable when combined with distributed tracing. A trace identifier can link application events generated by multiple services during the same request.

However, structured logging is not a substitute for good instrumentation. The recorded fields must be accurate, consistent and useful for the investigations engineers need to perform.

04 / Log Severity and Event Analysis

Understanding the Operational Event Stream

Log severity levels help distinguish normal application activity from conditions that may require attention. Common levels include DEBUG, INFO, WARN and ERROR.

Severity alone does not determine incident priority. A warning may indicate an emerging capacity problem, while an error may represent a transient failure that the application recovered from successfully.

Distributed Log Event Stream
ILLUSTRATIVE LOGS
14:32:08 INFO api-gateway Incoming POST /orders request accepted
14:32:08 INFO auth-service Request authentication completed
14:32:09 WARN order-service Database connection acquisition delayed
14:32:10 ERROR order-service Database connection timeout during create-order
14:32:10 ERROR api-gateway Upstream order request returned HTTP 503
Example only. These events illustrate how one downstream dependency problem may appear in several services.

In this example, the API gateway records the final failure, but the order service logs provide evidence of a database connection problem.

Searching only for API gateway errors would reveal the symptom without necessarily identifying the likely downstream cause.

05 / Trace and Log Correlation

Connecting Application Events to Distributed Requests

A trace ID provides a useful correlation key between distributed traces and application logs.

When the same trace identifier is included in relevant log events, engineers can search across services and reconstruct the event sequence associated with a specific request.

01
Identify the Request

Select an affected distributed trace or obtain a trace ID from an application error.

02
Search Correlated Logs

Filter centralised log records using the trace ID and inspect events across participating services.

03
Validate the Failure

Compare the event sequence with trace timing and metrics to identify the likely cause.

Correlation requires consistent instrumentation. Applications must preserve trace context and include the relevant identifiers in their logging configuration.

In asynchronous systems, correlation may also require message identifiers, workflow identifiers or other application-specific context. A single trace ID does not always represent an entire long-running business workflow.

06 / Centralised Logging Platforms

Collection, Storage and Investigation Technologies

Centralised logging architectures can be built using open-source components, commercial observability platforms or managed cloud services.

Different technologies focus on different parts of the pipeline, including collection, processing, storage, indexing, querying and visualisation.

Technology Typical Role Engineering Use
Fluent Bit Lightweight log collection and processing Forward container and host events
Fluentd Log collection and routing Transform and deliver log records
Logstash Event processing pipeline Parse, enrich and transform events
Elasticsearch Search and analytics storage Index and query log data
Kibana Log search and visualisation Explore events and build dashboards
Grafana Loki Log aggregation and querying Investigate logs using labels and queries
Managed Cloud Logging Hosted collection, storage and analysis Centralise cloud and application events

A common Elastic-based architecture combines collection agents, Logstash or other processing tools, Elasticsearch storage and Kibana visualisation.

Grafana Loki takes a different approach by indexing labels rather than the full contents of every log message, which influences how engineers design queries and choose label dimensions.

Platform selection should consider query requirements, data volume, retention, operational complexity, integration with traces and metrics, and total ownership cost.

07 / Log Governance

Managing Security, Retention and Logging Costs

Centralised logging can introduce substantial operational and security considerations. Large distributed applications may generate significant volumes of events, including records containing sensitive application or infrastructure information.

Logging strategies should therefore include deliberate controls for access, retention, data minimisation and event volume.

01
Data Protection

Avoid recording passwords, access tokens and unnecessary personal information. Apply appropriate redaction and access controls.

02
Retention Policies

Define retention periods based on operational needs, security requirements, compliance obligations and storage cost.

03
Volume Management

Use appropriate severity levels, filtering and sampling policies to reduce noise without losing essential investigation evidence.

Logging systems should also be monitored as production services. Collection failures, dropped events, storage saturation and delayed ingestion can create visibility gaps precisely when engineers need telemetry most.

08 / Engineering Case Study

Investigating an Order Processing Failure

Consider a microservices application where customers intermittently receive HTTP 503 responses when submitting orders.

The API gateway reports the failed requests, but its logs do not identify the underlying database condition.

Distributed Log Investigation
STEP 01 / DETECT Gateway Error

The API gateway reports HTTP 503 responses for order submission requests. Error metrics confirm an increasing failure rate.

STEP 02 / CORRELATE Shared Trace ID

Engineers search the centralised logging platform for the affected request's trace ID across services.

STEP 03 / INVESTIGATE Database Timeout

The order service records connection acquisition delays and database timeout errors.

Engineering outcome: Correlated logs point toward a database connection problem as the likely cause of the order failures. Engineers can validate the hypothesis using connection-pool metrics, database health and distributed trace timing.

The important distinction is that the API gateway recorded the visible symptom, while the downstream service recorded evidence of the likely cause.

Centralised logging made it possible to connect these events without manually accessing individual containers or hosts.

Network Insight · Engineering Principle

Logs become most valuable when they preserve context across distributed execution. Structured fields, consistent timestamps, trace correlation and centralised collection allow engineers to move from isolated error messages to evidence-based investigation. Logging quality should be designed into the application, not added only after an incident.

Section 05 — Key Engineering Takeaways
✓ Centralised logging collects distributed events into a searchable operational system.
✓ Structured logs improve consistency, filtering and investigation.
✓ Trace IDs connect application events across participating services and requests.
✓ Collection pipelines must account for buffering, processing, delivery and retention.
✓ Logging platforms support collection, storage, search and analysis through different architectures.
✓ Security, privacy and volume controls are essential for sustainable logging operations.
06
NEXT SECTION OpenTelemetry & Instrumentation
05 Interactive Engineering Lab

Centralised Logging Investigation Console

Investigate a distributed checkout application through structured logs. Search telemetry, filter services and severity levels, inspect JSON records, correlate trace identifiers and reconstruct the sequence of events leading to a production incident.

LOGGING HEALTHY
01 / Select a Production Incident
02 / Centralised Logging Pipeline
01
Application Structured events
02
Collector Forward & enrich
03
Storage Index & retain
04
Investigation Search & correlate
03 / Log Ingestion Metrics
Indexed Events 0
Error Events 0
Warning Events 0
Services Observed 0
04 / Searchable Central Log Console

Distributed Application Log Stream

0 EVENTS
NETWORK INSIGHT / LOG EXPLORER

Select any log record to inspect its complete structured fields. Search is case-insensitive and applies to message text, service, severity, trace ID and error code.

Structured JSON Inspector

EVENT DETAILS
Select a log event.

Log Correlation Engine

TRACE CONTEXT
INFO Events Normal application activity
0
WARN Events Degraded behaviour
0
ERROR Events Failed operations
0
Select a log record and press Correlate Trace to isolate related events.
05 / Reconstruct the Request Journey

Correlated Request Timeline

NO TRACE SELECTED
Select a log record
Correlation groups events that share a trace ID. This can help reconstruct a request across services, provided trace context has been propagated correctly.
06 / Engineering Investigation Challenge

Identify the Root Cause

Investigate the log stream, filter errors and warnings, inspect structured fields and identify the most likely cause of the selected incident.

Search the logs and select the diagnosis best supported by the evidence.

Investigation Activity

AUDIT TRAIL
Engineering principle: Centralised logging enables engineers to search events across distributed services. Structured fields, consistent timestamps, correlation identifiers and contextual error information make investigations more reliable. Log correlation provides evidence, but a matching trace ID alone does not establish causation. All incidents and records in this lab are simulated.
06 MICROSERVICES OBSERVABILITY · SECTION 06

OpenTelemetry & Instrumentation

Observability begins inside the application. Before metrics can be queried, traces can be visualised or logs can be correlated, software must generate meaningful telemetry describing its behaviour.

In a microservices architecture, instrumentation provides visibility into requests, operations, dependencies and failures across independently deployed components.

OpenTelemetry provides a vendor-neutral collection of APIs, SDKs, instrumentation libraries, semantic conventions and tools for generating, collecting and exporting observability data.

Rather than requiring every application to integrate directly with a particular monitoring platform, OpenTelemetry enables engineers to use standardised instrumentation and telemetry pipelines that can connect to different observability backends.

01 / Application Instrumentation

How Applications Produce Observability Data

Instrumentation introduces measurement and diagnostic information into application execution. It may be added automatically through supported frameworks and libraries, or manually through application code.

The goal is not simply to collect more data. Instrumentation should expose the operations, dependencies and failure conditions that engineers need to understand.

A
Automatic Instrumentation

Supported libraries and agents can instrument common frameworks, HTTP clients, database calls and messaging systems without requiring extensive changes to application code.

Framework-Level Visibility
M
Manual Instrumentation

Developers create custom spans, metrics and attributes to represent important business operations, application-specific workflows and internal processing stages.

Business Context
C
Context Propagation

Trace context is propagated between services so independently recorded operations can be associated with the same distributed request.

Cross-Service Correlation

Automatic instrumentation is useful for establishing broad coverage quickly. Manual instrumentation adds the application context that generic libraries cannot infer, such as a checkout validation stage or an order approval decision.

Both approaches are complementary. A production observability strategy often combines automatic framework instrumentation with carefully chosen application-specific measurements.

02 / OpenTelemetry Architecture

From Application SDKs to Observability Backends

OpenTelemetry separates telemetry generation from the systems used to store and analyse the resulting data.

Applications use instrumentation and SDKs to create telemetry. The data can be exported directly to a compatible backend or sent through an OpenTelemetry Collector for additional processing and routing.

OpenTelemetry Reference Architecture Conceptual Data Flow
API
API Gateway HTTP instrumentation and request spans
ORD
Order Service Custom spans and application metrics
PAY
Payment Service Dependency spans and error context
OTel
OpenTelemetry Collector

Receivers · Processors · Exporters

M
Metrics Backend Time-series analysis and dashboards
L
Logs Backend Searchable events and correlation
T
Traces Backend Distributed request investigation

Architecture note: The Collector is optional in some deployments, and the destination systems may use different protocols or storage models. The diagram illustrates a common centralised collection pattern.

OpenTelemetry commonly uses the OpenTelemetry Protocol (OTLP) to transport telemetry between SDKs, Collectors and compatible receivers.

OTLP supports different transport options, including gRPC and HTTP. The protocol transports telemetry; it does not itself provide a long-term metrics, logs or traces storage system.

03 / The OpenTelemetry Collector

Building a Flexible Telemetry Processing Pipeline

The OpenTelemetry Collector is a configurable telemetry service that receives, processes and exports observability data.

It can run as a central gateway, as an agent close to applications, or in architectures combining both deployment patterns.

01
Receivers

Accept telemetry from applications and other systems through supported protocols or collection mechanisms.

02
Processors

Apply configured operations such as batching, memory protection, filtering or attribute modification.

03
Exporters

Send processed telemetry to downstream Collectors or compatible observability platforms and services.

Collector pipelines are configured by telemetry signal. Metrics, logs and traces can have different processing requirements and destination systems.

For example, a production pipeline may batch trace data before export, filter unnecessary log events and attach environment metadata to telemetry records.

Collector Pipeline Example YAML · Illustrative Configuration
# Minimal OTLP traces pipeline receivers: otlp: protocols: grpc: endpoint: 0.0.0.0:4317 processors: batch: {} exporters: otlp: endpoint: traces-backend:4317 tls: insecure: true service: pipelines: traces: receivers: [otlp] processors: [batch] exporters: [otlp]

This simplified example accepts OTLP trace data, batches it and forwards it to another OTLP-compatible endpoint.

Security consideration: The insecure transport setting shown here is appropriate only for a controlled demonstration environment. Production deployments should use appropriate TLS, authentication, network restrictions and secret management.

04 / Distributed Context Propagation

Preserving Request Identity Across Services

A distributed trace can span multiple independently deployed services. To reconstruct the request journey, each participating service needs sufficient context to associate its spans with the same trace.

The W3C Trace Context standard defines HTTP headers such as traceparent and tracestate for carrying trace context between components.

Example: Context Propagation
API Gateway Receives request and creates span
Order Service Extracts context and creates child span
Payment Service Continues the distributed trace
traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736- 00f067aa0ba902b7-01

The trace ID identifies the overall trace. The parent span identifier and trace flags provide additional context used to connect spans and communicate sampling decisions.

Context propagation is particularly important when requests pass through API gateways, service meshes, message brokers or asynchronous workers.

If context is not propagated across a service boundary, the downstream operation may appear as an unrelated trace, reducing the completeness of the investigation.

05 / Manual Instrumentation

Adding Business Context to Distributed Traces

Automatic instrumentation can record common operations such as HTTP requests and database queries. However, it may not understand the business meaning of those operations.

Manual instrumentation allows developers to create spans around important application logic and attach attributes that improve troubleshooting.

Custom Application Span JavaScript · OpenTelemetry API
const { trace } = require('@opentelemetry/api'); const tracer = trace.getTracer('order-service'); async function createOrder(order) { return tracer.startActiveSpan( 'order.create', async (span) => { try { span.setAttribute( 'order.type', order.type ); return await saveOrder(order); } catch (error) { span.recordException(error); throw error; } finally { span.end(); } } ); }

This example creates a span for an order-processing operation, records a business attribute and captures exceptions before closing the span.

A functioning deployment also requires an appropriate OpenTelemetry SDK, span processor, exporter and provider configuration. The API code alone does not guarantee that spans will be exported.

Engineers should avoid attaching sensitive data or unbounded high-cardinality values to spans without considering privacy, storage and analysis requirements.

06 / Sampling and Telemetry Volume

Balancing Diagnostic Detail and Operational Cost

High-volume microservices can generate large amounts of trace data. Recording and exporting every span may be expensive or unnecessary for some environments.

Sampling strategies allow engineers to control the amount of trace telemetry retained while preserving useful diagnostic information.

H
Head-Based Sampling

A sampling decision is made near the beginning of a trace, often using a configured probability or parent sampling decision.

Benefit: Sampling can reduce telemetry processing and export volume early in the request lifecycle.

T
Tail-Based Sampling

A sampling decision is made after enough trace data has been collected to evaluate characteristics such as errors or latency.

Benefit: Sampling policies can retain traces associated with significant failures or slow operations.

Tail-based sampling requires additional buffering and processing capacity because the decision is delayed until sufficient trace information becomes available.

Sampling can also introduce incomplete visibility if decisions or context propagation are not coordinated across services. Sampling policies should therefore be tested against real investigation requirements.

07 / Instrumentation Design Considerations

Avoiding Common Observability Design Mistakes

Instrumentation quality matters more than the raw quantity of telemetry. Excessive instrumentation can create unnecessary operational overhead, while incomplete instrumentation leaves critical diagnostic gaps.

Design Area Potential Problem Recommended Approach
Span Boundaries Missing important business operations Instrument meaningful application stages
Context Propagation Disconnected distributed traces Preserve trace context across service boundaries
Attributes Unbounded values or sensitive information Use deliberate, privacy-aware attributes
Sampling Important incidents are not retained Align sampling policies with diagnostic needs
Collector Reliability Telemetry loss during pipeline failures Monitor queues, memory, retries and export errors
Service Identity Telemetry cannot be attributed correctly Apply consistent resource attributes such as service.name

Consistent service identity is especially important in environments where workloads are frequently redeployed or rescheduled.

OpenTelemetry resource attributes provide information about the entity producing telemetry, helping observability systems group data by service, deployment and infrastructure context.

08 / Engineering Case Study

Finding a Missing Segment in a Distributed Trace

Consider a checkout request that passes through an API gateway, order service and payment service. Engineers can see the gateway and order spans, but the payment operation is missing from the distributed trace.

OpenTelemetry Investigation
STEP 01 / OBSERVE Incomplete Trace

The trace visualisation ends at the order service even though application logs confirm that payment processing occurred.

STEP 02 / INSPECT Context Boundary

Engineers inspect the outbound payment request, inbound context extraction and service instrumentation.

STEP 03 / VALIDATE Propagation Fix

After correcting the identified context issue, new test requests produce correlated payment spans.

Engineering outcome: The investigation demonstrates how a context propagation problem can reduce distributed trace completeness. Engineers should also check sampling decisions, exporter health and Collector processing before concluding that propagation is the cause.

A missing span does not necessarily mean the underlying operation did not occur. It may indicate a gap in instrumentation, sampling, context propagation or telemetry delivery.

This distinction is important when using traces to investigate complex distributed failures.

Network Insight · Engineering Principle

Observability is engineered at the point where applications generate meaningful telemetry. OpenTelemetry provides a vendor-neutral instrumentation framework, while context propagation, Collector pipelines and sampling policies determine how effectively that telemetry supports real investigations.

Section 06 — Key Engineering Takeaways
✓ OpenTelemetry provides vendor-neutral APIs, SDKs and tools for observability.
✓ Automatic and manual instrumentation provide complementary visibility.
✓ The OpenTelemetry Collector receives, processes and exports telemetry data.
✓ W3C Trace Context supports distributed trace correlation across service boundaries.
✓ Sampling policies balance diagnostic visibility with telemetry cost.
✓ Consistent resource attributes and reliable telemetry pipelines are essential for operational visibility.
07
NEXT SECTION Kubernetes & Cloud-Native Monitoring
06 Interactive Engineering Lab

OpenTelemetry Pipeline Builder

Configure an end-to-end observability pipeline. Enable application instrumentation, control telemetry processing, simulate exporter failures and investigate how collector configuration affects the delivery of metrics, logs and traces.

PIPELINE HEALTHY
01 / Select an Engineering Scenario
02 / OpenTelemetry Architecture Explorer
Metrics Logs Traces

Instrumentation & Collector Configuration

PIPELINE CONTROLS
Enabled Telemetry Signals
Configure the pipeline and generate telemetry to observe how each setting changes delivery.

Pipeline Throughput

SIGNAL DELIVERY
Metrics 0 / 0
Logs 0 / 0
Traces 0 / 0
Bars represent the cumulative proportion of generated signal records successfully delivered to the simulated backend.
03 / Collector Telemetry
Generated Records 0
Exported Records 0
Dropped / Filtered 0
Export Failures 0
04 / Collector Configuration Preview

Collector Configuration

ILLUSTRATIVE YAML


   

This is a teaching-oriented configuration preview. Component availability and valid settings depend on the installed Collector distribution and version.

Collector Diagnostic Console

EVENT STREAM
NETWORK INSIGHT / OTEL COLLECTOR
The collector is ready. Generate telemetry to inspect simulated ingestion, processing and export events.
05 / Engineering Investigation Challenge

Identify the Pipeline Condition

Examine the configuration, delivery statistics and diagnostic console. Select the diagnosis best supported by the active scenario.

Inspect the pipeline before selecting the most likely engineering diagnosis.
Engineering principle: OpenTelemetry provides APIs, SDKs, semantic conventions and a Collector ecosystem for generating and processing observability data. Receiver, processor and exporter configuration determines how telemetry travels through the pipeline. Sampling, filtering, batching and backend availability affect what engineers can observe. This lab uses simplified telemetry counts and simulated failures; it does not execute a real OpenTelemetry Collector.
07 MICROSERVICES OBSERVABILITY · SECTION 07

Kubernetes & Cloud-Native Monitoring

Kubernetes introduces a dynamic execution environment in which application workloads can be scheduled, restarted, replaced and scaled across multiple infrastructure nodes. Effective monitoring must therefore connect application behaviour with the changing state of the orchestration platform.

In a traditional environment, an application might run on a relatively stable collection of servers. Engineers can often associate performance problems with known hosts and predictable infrastructure dependencies.

Kubernetes changes that operational model. Pods can be recreated, workloads can move between nodes, and the number of application replicas can increase or decrease in response to demand.

Cloud-native monitoring must observe both the application and the platform responsible for running it. A healthy Kubernetes cluster does not automatically imply a healthy application, and an application failure does not necessarily originate inside application code.

01 / Kubernetes Observability Architecture

Monitoring Applications Across a Dynamic Cluster

A Kubernetes observability architecture typically collects information from application containers, pods, nodes, Kubernetes control-plane components and supporting infrastructure.

These signals may be gathered through application instrumentation, Kubernetes APIs, metrics endpoints, container log collectors and infrastructure monitoring agents.

Kubernetes Monitoring Architecture Conceptual Cluster
Kubernetes Cluster · Application Workloads
Worker Node 01 Ready
API
API Gateway Pod · Running
AU
Auth Service Pod · Running
Kubelet · Container Runtime
Worker Node 02 Ready
ORD
Order Service Pod · Running
DB
Database Client Pod · Restarting
Kubelet · Container Runtime
Worker Node 03 Ready
PAY
Payment Service Pod · Running
WRK
Async Worker Pod · Running
Kubelet · Container Runtime
OTel
Telemetry Collection Layer Application instrumentation · Metrics scraping · Log collection · Kubernetes metadata
M
Metrics Resource utilisation, latency and error rates
L
Logs Container events, failures and exceptions
T
Traces Distributed requests and dependency timing

Architecture note: The illustrated collection layer represents multiple possible components, not necessarily a single agent. Real deployments may combine Prometheus, OpenTelemetry Collectors, Kubernetes exporters and cloud-managed services.

Kubernetes metadata provides important context for telemetry. Labels such as namespace, workload identity, pod name and node name help engineers determine where a signal originated.

Because individual pod identities can change frequently, long-term monitoring should also aggregate telemetry by stable workload and service dimensions.

02 / The Four Observability Layers

Connecting Application Health to Infrastructure State

Effective Kubernetes monitoring requires visibility across several operational layers. Each layer answers a different set of troubleshooting questions.

01
Application

Request latency, error rates, throughput, business operations, distributed traces and application exceptions.

02
Container & Pod

Restarts, readiness, resource consumption, container termination reasons and pod scheduling state.

03
Node & Cluster

CPU and memory pressure, node availability, storage health, scheduling capacity and orchestration events.

04
Network & Dependencies

Service connectivity, DNS resolution, network policies, ingress traffic and external service availability.

These layers are interconnected. A request latency increase may originate from application logic, but it may also result from CPU throttling, an overloaded dependency, storage delays or network disruption.

Engineers should correlate signals between layers rather than assuming that the component reporting an error is responsible for causing it.

03 / Ephemeral Pods and Workload Lifecycle

Monitoring Workloads That Continuously Change

Kubernetes controllers manage the desired state of application workloads. If a pod becomes unavailable or a deployment changes, the platform may create replacement pods.

Monitoring must account for these transitions because the disappearance of a pod does not necessarily indicate that its workload has stopped serving requests.

Illustrative Pod Replacement Journey
01
Pod Created Workload controller requests a pod
02
Scheduled Scheduler selects an eligible node
03
Running Container starts and becomes ready
04
Terminated Pod removed during failure or rollout
05
Replacement Controller creates another pod

This is a simplified workload lifecycle illustration rather than the complete Kubernetes pod phase state machine. Real workloads can experience additional states, retries and termination paths.

A critical operational distinction exists between pod phase, container state and application readiness.

A pod may be in the Running phase while its application is not ready to accept traffic. Similarly, a container may repeatedly restart without the underlying node becoming unavailable.

Monitoring should therefore include readiness conditions, restart counts, termination reasons and workload availability rather than relying only on pod phase.

04 / Kubernetes Monitoring Signals

Metrics That Reveal Workload and Cluster Problems

Kubernetes exposes several categories of operational signals. Infrastructure metrics help identify resource pressure, while application metrics reveal the effect of that pressure on user-facing services.

CPU
CPU & Memory

Monitor CPU utilisation, CPU throttling, memory working sets, memory limits and out-of-memory termination events.

POD
Pod & Workload Health

Monitor desired versus available replicas, restart counts, readiness conditions and scheduling failures.

NET
Network & Storage

Monitor network errors, packet drops where exposed, DNS failures, storage latency and volume availability.

A common monitoring architecture uses Prometheus-compatible metrics endpoints and exporters to collect infrastructure and application measurements.

Tools such as kube-state-metrics expose Kubernetes object-state information, while kubelet and container-related metrics provide additional resource and runtime visibility.

Infrastructure metrics should be interpreted alongside application service-level indicators. High CPU utilisation is not automatically an incident, but high CPU combined with increased p99 latency and request errors may indicate a meaningful performance problem.

05 / Kubernetes Networking Observability

Following Requests Through the Cluster Network

Kubernetes networking introduces several layers between a client and the application container. Depending on the deployment, requests may pass through an external load balancer, ingress controller or Gateway API implementation, Kubernetes Service and the cluster networking layer.

Example Application Traffic Journey
Client External request
Ingress / Gateway Entry point and routing
Kubernetes Service Stable service abstraction
Application Pod Selected backend endpoint

Network investigation: A failed request can result from routing configuration, endpoint availability, DNS resolution, network policy enforcement, backend readiness or application failure.

Kubernetes Services provide stable network abstractions for changing groups of backend pods. EndpointSlice objects describe the endpoints associated with Services.

When an application becomes unavailable, engineers should verify whether suitable ready endpoints exist before assuming that the problem is caused by the external network.

NetworkPolicy objects may also influence connectivity when the installed network plugin supports their enforcement.

For advanced troubleshooting, network-level telemetry can complement application traces by exposing connection attempts, policy decisions, retransmissions or packet drops where supported by the underlying infrastructure.

06 / Kubernetes Troubleshooting Commands

Combining CLI Investigation with Observability Data

Observability dashboards help engineers detect patterns, but Kubernetes command-line inspection remains useful for validating the current state of workloads.

The following commands provide examples of operational checks during a service investigation.

Kubernetes Investigation Commands kubectl · Read-Only Examples
# Inspect application pods kubectl get pods -n production -o wide # Examine a failing pod and its events kubectl describe pod order-pod -n production # Inspect recent container logs kubectl logs order-pod -n production --tail=100 # Inspect logs from a previous container instance kubectl logs order-pod -n production --previous # Inspect deployment availability kubectl get deployments -n production # Inspect Service endpoint discovery kubectl get endpointslices -n production # Inspect recent namespace events kubectl get events -n production --sort-by=.metadata.creationTimestamp

The pod name in these commands is illustrative and must be replaced with the actual workload identifier. The --previous option is useful when a previous container instance is available, but it does not provide an unlimited historical log archive.

Commands reveal current or recently retained Kubernetes state. Centralised metrics, logs and traces provide the historical context needed to compare conditions before, during and after an incident.

07 / Cloud-Native Monitoring Platforms

Integrating Kubernetes with Managed Observability

Kubernetes monitoring can be implemented through self-managed observability systems or managed cloud platforms.

Cloud providers offer monitoring, logging and tracing integrations for managed Kubernetes environments, although collection coverage, configuration and pricing vary.

Platform / Tool Typical Role Operational Value
Prometheus Metrics collection and querying Application and infrastructure metrics
Grafana Observability dashboards and exploration Visualise and correlate operational signals
kube-state-metrics Kubernetes object-state metrics Workload availability and object conditions
OpenTelemetry Instrumentation and telemetry pipelines Standardise collection and export
Google Cloud Observability Managed monitoring, logging and tracing Integrate GKE and Google Cloud telemetry
Amazon CloudWatch AWS monitoring and log analysis Observe AWS and EKS environments
Azure Monitor Azure monitoring and analytics Observe Azure and AKS workloads

For example, Google Kubernetes Engine can integrate with Google Cloud Monitoring and Cloud Logging, while Cloud Trace supports distributed tracing investigations.

Cloud network telemetry, such as VPC Flow Logs where supported and enabled, may provide additional information about traffic patterns and network behaviour. It should not be treated as a replacement for packet-level investigation or application tracing.

Engineers should also monitor the telemetry pipeline itself, including ingestion delays, dropped records, collection errors and exporter health.

08 / Engineering Case Study

Investigating a Kubernetes Application Failure

Consider an order-processing application deployed across multiple Kubernetes worker nodes. Customers begin experiencing intermittent HTTP 503 responses shortly after a deployment update.

The cluster nodes remain Ready, but application availability has deteriorated.

Kubernetes Incident Investigation
STEP 01 / DETECT Application Errors

Request metrics reveal increasing HTTP 503 errors. Distributed traces identify the order service as the affected application path.

STEP 02 / CORRELATE Pod Availability

Kubernetes metrics show reduced ready replicas. Pod events reveal readiness probe failures following the deployment update.

STEP 03 / INVESTIGATE Configuration Change

Engineers compare the deployment configuration and application startup behaviour to identify why new pods fail readiness.

Engineering outcome: The evidence points toward a readiness or startup problem introduced during the rollout. Engineers can validate the hypothesis by inspecting probe configuration, application logs, rollout history and available Service endpoints before applying a corrective change.

The important lesson is that Kubernetes infrastructure can remain operational while the application becomes unavailable.

Node readiness alone does not guarantee sufficient application capacity, correct Service endpoint selection or successful end-to-end request processing.

By correlating application metrics, traces, logs and Kubernetes state, engineers can distinguish orchestration problems from application and dependency failures.

Network Insight · Engineering Principle

Cloud-native observability requires correlation across application, orchestration, infrastructure and network layers. Kubernetes can maintain the desired workload state, but engineers must still verify that the application is ready, reachable and delivering the expected service-level behaviour. The most useful monitoring systems connect workload identity, infrastructure conditions and application outcomes into a unified investigation.

Section 07 — Key Engineering Takeaways
✓ Kubernetes workloads are dynamic, so monitoring must account for ephemeral pods and changing replicas.
✓ Application health and Kubernetes infrastructure health are related but distinct.
✓ Pod readiness, restart counts, resource pressure and workload availability provide essential troubleshooting signals.
✓ Kubernetes networking introduces Service, endpoint, ingress, DNS and policy considerations.
✓ Prometheus, OpenTelemetry and cloud-managed monitoring platforms can provide complementary observability capabilities.
✓ Effective incident investigation correlates metrics, logs, traces and Kubernetes object state.
08
NEXT SECTION Service Mesh, APM & Dependency Mapping
07 Interactive Engineering Lab

Kubernetes Monitoring Explorer

Investigate a cloud-native application deployed across three Kubernetes worker nodes. Inspect pod lifecycle states, resource utilisation, deployment readiness, simulated cluster events and kubectl diagnostics to identify production incidents.

CLUSTER HEALTHY
01 / Select a Kubernetes Incident
02 / Cluster Health Overview
Worker Nodes 3
Ready Pods 0
Unavailable Pods 0
CPU Utilisation 0%
Memory Utilisation 0%
03 / Interactive Kubernetes Cluster
Production Cluster / namespace: shop 3 WORKER NODES

Select any pod to inspect its lifecycle state, restart count, resource usage and diagnostic context. Select a worker node heading to inspect node resources.

Resource & Workload Inspector

POD DETAILS
Select a pod or node to inspect its state.

Deployment Readiness

REPLICAS
Engineering insight: A running pod is not necessarily ready to serve traffic. Readiness probes determine whether a pod should be included in matching Service endpoints.
04 / Kubernetes Diagnostic Terminal

Simulated kubectl Investigation

READ-ONLY COMMANDS
NETWORK INSIGHT / KUBERNETES CLI
Select a command to inspect the simulated cluster.

Commands operate on the lab's simulated state. No connection to a real Kubernetes cluster is made.

05 / Kubernetes Event Timeline

Cluster Events

CONTROL PLANE EVENTS
06 / Engineering Investigation Challenge

Identify the Root Cause

Inspect pod states, restart counts, resource metrics, deployment readiness and kubectl output. Select the diagnosis that best matches the evidence.

Investigate the cluster and select the most likely cause.
Engineering principle: Kubernetes observability requires correlating workload state, scheduling decisions, container resource usage, probe results and cluster events. Pod phases, container waiting reasons and termination reasons are different concepts: CrashLoopBackOff is a container waiting reason, while OOMKilled is a termination reason. This lab models those distinctions in a simplified environment. All cluster data and kubectl output are simulated.
08 MICROSERVICES OBSERVABILITY · SECTION 08

Service Mesh, APM & Dependency Mapping

In a distributed microservices environment, application performance depends on more than the behaviour of individual services. It also depends on the communication paths, traffic policies and downstream dependencies connecting those services.

A service may appear healthy when examined in isolation while requests passing through it experience high latency, repeated retries or failures in downstream systems.

Service mesh technologies, application performance monitoring platforms and service dependency maps provide complementary ways to understand these relationships.

The objective is to move from monitoring individual components to observing the behaviour of the complete application system.

01 / Service Mesh Architecture

Introducing a Dedicated Service Communication Layer

A service mesh provides infrastructure capabilities for managing and observing communication between application services. Depending on the implementation, these capabilities may include traffic routing, encryption, identity enforcement, retries, timeouts and telemetry generation.

In a traditional sidecar-based architecture, a proxy is deployed alongside each participating application workload. The proxy handles selected inbound and outbound service traffic.

A control plane distributes configuration and policy to the data-plane proxies, allowing traffic behaviour to be managed consistently across many services.

Sidecar-Based Service Mesh Architecture Conceptual Model
CP
Service Mesh Control Plane Distributes routing, security and proxy configuration
API
API Gateway Application Container
P
Sidecar Proxy Traffic + Telemetry
ORD
Order Service Application Container
P
Sidecar Proxy Traffic + Telemetry
PAY
Payment Service Application Container
P
Sidecar Proxy Traffic + Telemetry

Architecture note: This diagram illustrates a sidecar deployment model. Some modern service mesh architectures use shared node-level components and optional specialised proxies instead of deploying a sidecar beside every application. The animation represents logical service communication, not a literal packet path through the diagram.

Examples of service mesh technologies include Istio and Linkerd, although their architectures, features and operational requirements differ.

Service mesh observability can provide request rates, response codes, latency measurements and information about traffic between participating workloads.

However, mesh telemetry does not automatically reveal every internal application operation. Database processing, business logic and asynchronous workflows may still require application instrumentation.

02 / Service Mesh Capabilities

Traffic Management, Security and Observability

Service meshes are frequently introduced to provide consistent communication controls without requiring every application team to implement those controls independently.

TM
Traffic Management

Configure routing behaviour, retries, timeouts, traffic splitting and other supported communication policies.

SEC
Service Security

Support authenticated service communication, mutual TLS and identity-based policy enforcement.

OBS
Traffic Observability

Generate network and application-protocol telemetry that helps engineers inspect communication between services.

Traffic management policies require careful design. Retries may improve resilience during transient failures, but uncontrolled retries can amplify load on an already overloaded dependency.

Similarly, a timeout configured at one service boundary should be considered alongside the overall request deadline and the timeouts of downstream calls.

03 / Application Performance Monitoring

Understanding Application Behaviour Beyond the Network

Application Performance Monitoring (APM) focuses on the performance and reliability of application transactions, operations and dependencies.

APM platforms commonly combine application instrumentation, transaction traces, service-level metrics and dependency information to help engineers understand where application time is spent.

For example, an order-processing transaction might include request validation, inventory lookup, payment authorisation and database updates.

Each operation can contribute to the observed response time. Instrumentation allows engineers to identify slow operations, error-producing dependencies and changes in transaction performance.

Technology Primary Visibility Typical Limitation
Service Mesh Inter-service communication, traffic policies and proxy-observed metrics Limited visibility into application internals without instrumentation
APM Transactions, application operations and dependency performance Coverage depends on supported instrumentation and configuration
Distributed Tracing Individual request journeys and span timing Missing context or sampling can leave incomplete traces
Infrastructure Monitoring CPU, memory, network, storage and platform health Resource metrics alone cannot explain application behaviour

These technologies should not be treated as mutually exclusive. APM may use distributed tracing internally, while service mesh telemetry can enrich dependency maps and application dashboards.

04 / Service Dependency Mapping

Visualising the Relationships Between Microservices

A service dependency map represents the communication relationships between application components. Nodes typically represent services, while edges represent observed requests or other dependency relationships.

Dependency maps help engineers understand which components communicate, identify shared dependencies and assess the potential impact of failures.

Example Microservices Dependency Map Illustrative Incident
Observed Dependency
Degraded Path
Investigate

Interpretation: The payment service is marked as degraded, while the shared database remains a possible contributing dependency. A dependency map identifies relationships and symptoms; it does not establish root cause by itself.

A useful dependency map may enrich edges with request volume, error rates and latency measurements.

Engineers can then identify whether a degraded dependency is isolated to one service or affects multiple consumers.

Dependency maps also support change-impact analysis. Before modifying a shared service, teams can inspect which applications rely on it.

05 / Latency Attribution

Identifying Which Dependency Contributes to Delay

An application transaction may involve multiple downstream operations. APM and distributed tracing help engineers determine which operations contribute most significantly to the observed response time.

Consider an illustrative checkout transaction with several sequential processing stages.

Example Sequential Operation Durations
Gateway
40 ms
Order Logic
80 ms
Inventory
120 ms
Payment
500 ms
Database
60 ms

Illustrative timing: If these operations are strictly sequential and non-overlapping, their combined duration is 800 ms. In real traces, parent-child spans and parallel operations often overlap, so span durations must not simply be added together. A trace waterfall and critical-path analysis are needed to attribute end-to-end latency correctly.

In this example, the payment operation represents the largest individual delay. However, the next investigation should determine whether that delay originates in payment processing, network communication, retries or another downstream dependency.

APM systems can help distinguish application execution time from waiting on external calls, provided the necessary instrumentation is available.

06 / Service Mesh Security and Reliability

Observing Encryption, Identity and Traffic Policies

Service mesh technology can introduce security and reliability controls into service-to-service communication.

These controls are operationally important because policy changes may affect application connectivity even when application code remains unchanged.

TLS
Mutual TLS

Supported meshes can authenticate participating workloads and encrypt service communication. Certificate or identity problems may prevent connections from succeeding.

POL
Authorisation Policies

Identity-aware policies can restrict which services communicate. Engineers should inspect policy decisions when requests are unexpectedly denied.

REL
Retries & Timeouts

Reliability policies can influence request behaviour. Excessive retries or poorly chosen timeouts can increase latency and amplify failures.

Security and traffic telemetry should be correlated with application errors. A connection failure may originate from a policy decision, an identity problem or an unavailable destination.

Engineers should also distinguish proxy-level observations from application-level outcomes. A successful network connection does not guarantee that a business transaction completed successfully.

07 / Correlating Service Mesh and APM Data

Building a Unified Investigation View

The greatest value emerges when service communication telemetry, application instrumentation and infrastructure monitoring are analysed together.

For example, a service mesh may report increasing retries between two services. APM may reveal that the destination service is spending more time waiting on a database.

Kubernetes metrics may then reveal resource pressure affecting the database client or application workload.

Signal Investigation Question Potential Evidence
Mesh Metrics Is communication between services degrading? Error rates, retries, latency and traffic volume
APM Which application operation is experiencing delay? Transaction breakdown and operation timing
Distributed Traces Where did a specific request spend time? Span hierarchy, timing and exceptions
Kubernetes Metrics Is the workload resource constrained? CPU throttling, memory pressure and pod readiness
Application Logs What error or event occurred internally? Exceptions, timeouts and dependency failures

Consistent service naming, environment metadata and trace context propagation improve the ability to correlate these signals.

Without consistent identifiers, the same workload may appear under different names across monitoring systems, making investigations unnecessarily difficult.

08 / Engineering Case Study

Investigating a Degraded Payment Dependency

Consider a microservices checkout platform where customers report slow payment confirmation. The API gateway remains available, and the order service reports no obvious infrastructure failure.

However, end-to-end transaction latency has increased significantly.

Service Dependency Investigation
STEP 01 / DETECT Dependency Degradation

Service mesh metrics reveal increased latency and retries between the order service and payment service.

STEP 02 / CORRELATE Trace Investigation

Distributed traces identify long payment operations. APM highlights increased waiting time associated with a downstream call.

STEP 03 / VALIDATE Downstream Evidence

Engineers inspect payment logs, database metrics, timeout settings and retry behaviour to establish the actual source of delay.

Engineering outcome: The combined evidence narrows the investigation from a general checkout performance problem to the payment dependency path. Root cause still requires validation: the underlying issue could involve database latency, service saturation, network disruption or retry amplification.

Dependency mapping is particularly valuable in this situation because it identifies the services affected by a degraded component.

A shared dependency may create correlated failures across several otherwise healthy application services.

Engineers should therefore investigate both the failing component and its upstream and downstream relationships.

Network Insight · Engineering Principle

Distributed application performance is a property of the entire dependency graph, not merely the health of individual services. Service mesh telemetry exposes communication behaviour, APM reveals application transaction performance, and dependency maps help engineers understand the relationships between components. Correlating these views enables more effective troubleshooting and change-impact analysis.

Section 08 — Key Engineering Takeaways
✓ Service meshes provide infrastructure-level communication controls and traffic observability.
✓ APM provides visibility into application transactions, operations and dependencies.
✓ Dependency maps reveal communication relationships and potential failure propagation paths.
✓ Distributed trace timing must account for parallel operations and overlapping spans.
✓ mTLS, identity policies, retries and timeouts can influence service availability and latency.
✓ Effective investigations correlate mesh telemetry, APM, distributed traces, application logs and infrastructure metrics.
09
NEXT SECTION Incident Investigation, AIOps & Automation
08 Interactive Engineering Lab

Service Mesh Dependency Explorer

Investigate how requests travel between microservices. Explore application performance monitoring, service dependencies, proxy telemetry, mTLS, retries and circuit breakers while diagnosing latency and failures across a distributed application.

MESH HEALTHY
01 / Select an Engineering Scenario
02 / Application Performance Overview
Requests Observed 0
Last Request Latency —
Failed Requests 0
Circuit State Closed
03 / Interactive Service Dependency Map
Online Shop / Service-to-Service Traffic SIMULATED MESH
Healthy service Degraded latency Failure / blocked Connections represent logical dependencies.

Service Dependency Inspector

SELECT A SERVICE
Select a service in the topology to inspect its dependencies and performance.

Service Mesh Traffic Policies

PROXY CONTROLS
mTLS is enabled, one retry is configured and circuit-breaker protection is active.
04 / Distributed Request Trace

Request Critical Path

NO TRACE

Run a request trace to inspect service latency.

Trace spans will show where request time is spent and which dependency contributes most to the end-to-end latency.
05 / Service Mesh Event Timeline

Proxy & Application Events

TELEMETRY STREAM
06 / Engineering Investigation Challenge

Identify the Service Mesh Condition

Examine service dependencies, request latency, failures, traffic policies and trace spans. Select the diagnosis supported by the evidence.

Engineering principle: Service meshes can provide traffic management, identity-based transport security and proxy-level telemetry. APM and distributed tracing help connect dependency-level behaviour to application outcomes. Retries, timeouts and circuit breakers can improve resilience, but incorrect policies can amplify load, increase latency or hide underlying failures. This lab uses illustrative traffic and does not connect to a real service mesh.
09 MICROSERVICES OBSERVABILITY · SECTION 09

Incident Investigation, AIOps & Automation

Observability becomes operationally valuable when engineers can use telemetry to detect incidents, understand their impact, investigate contributing factors and restore reliable service.

Modern microservices environments generate enormous quantities of metrics, logs, traces, infrastructure events and dependency information. Collecting these signals is only the beginning of an effective incident-management process.

Engineers must determine which signals are meaningful, identify relationships between apparently separate failures and distinguish symptoms from underlying causes.

The objective of incident investigation is not simply to identify a failing service. It is to establish what changed, why the system behaved differently and how normal service can be restored safely.

01 / Incident Response Lifecycle

From Detection to Continuous Improvement

A structured incident-response lifecycle reduces uncertainty during production failures. It establishes a repeatable approach for detecting problems, investigating evidence and validating recovery.

Microservices Incident Response Lifecycle Operational Workflow
01
Detect Identify degraded service-level indicators
02
Triage Determine impact, scope and severity
03
Investigate Correlate telemetry and test hypotheses
04
Mitigate Restore service using controlled changes
05
Learn Review causes, improve controls and prevent recurrence

Operational note: Incident response is iterative. Teams may investigate while mitigation is underway and revisit earlier hypotheses when new evidence becomes available. Learning and prevention continue after immediate service recovery.

Detection should focus on meaningful service degradation rather than every individual infrastructure event.

For example, a temporary pod restart may not affect users if sufficient healthy replicas remain available. In contrast, a sustained increase in checkout failures represents a direct service-level problem.

Incident severity should reflect business impact, affected users, service criticality and operational risk.

02 / Detecting Meaningful Incidents

Moving from Infrastructure Alerts to Service-Level Signals

Traditional monitoring systems often generate alerts from infrastructure thresholds such as CPU utilisation, memory consumption or network interface errors.

These measurements remain important, but they do not always describe the experience of application users.

Microservices incident detection should therefore incorporate service-level indicators and application behaviour.

LAT
Latency

Detect sustained increases in request duration, especially at important percentiles and user-facing transaction boundaries.

ERR
Error Rates

Identify failed requests, dependency timeouts, exceptions and unsuccessful business transactions.

SAT
Saturation

Detect resource pressure, queue growth, connection exhaustion and other capacity constraints.

DEP
Dependencies

Observe downstream availability, retry amplification and cascading failures.

Service Level Objectives (SLOs) provide a useful foundation for alerting. Instead of reacting to every isolated measurement, teams can monitor whether a service is consuming its permitted error budget too quickly.

Multi-window, multi-burn-rate alerting can help distinguish severe short-lived degradation from slower persistent problems.

Alert thresholds and evaluation windows should be selected according to the service's reliability objectives, traffic patterns and operational requirements.

03 / Correlating Observability Signals

Building a Unified Incident Investigation

A production incident may generate many apparently unrelated alerts. An overloaded dependency can produce timeouts, retries, queue growth, application exceptions and infrastructure warnings.

Without correlation, each symptom may appear to represent a separate problem.

Observability Signal Correlation Conceptual Architecture
Metrics Latency · Errors
Logs Exceptions · Events
Traces Request Journeys
Platform Kubernetes · Mesh
COR
Incident Correlation & Analysis Time alignment · Service identity · Dependency relationships · Change events
01
Incident Scope Affected services and user impact
02
Hypotheses Possible causes supported by evidence
03
Response Controlled mitigation and validation

Important distinction: Correlation identifies related events and narrows the investigation. It does not automatically establish causation. Root-cause conclusions should be validated against independent evidence.

Consistent timestamps, service identifiers, deployment metadata and trace context improve correlation accuracy.

A useful investigation view should also include recent changes, such as deployments, configuration updates, policy modifications and infrastructure maintenance.

04 / Incident Timeline Reconstruction

Understanding the Sequence of Failure

A reliable incident investigation reconstructs what happened before, during and after service degradation.

The sequence of events can help engineers distinguish a triggering change from secondary symptoms.

Illustrative Production Incident Timeline
09:00
Deployment Begins

A new order-service application version begins rolling out across Kubernetes pods.

09:03
Latency Increases

Checkout p99 latency rises while the application remains partially available.

09:05
Error Rate Alert

Failed checkout requests increase, triggering a service-level reliability alert.

09:08
Dependency Identified

Traces reveal prolonged payment calls. Service mesh telemetry shows increased retries.

09:15
Mitigation Applied

Engineers apply a reviewed corrective action and monitor recovery of the affected transaction.

09:22
Service Recovery Validated

Error rates and transaction latency return to expected operating ranges.

Timeline reconstruction should account for clock differences, telemetry ingestion delays and incomplete records.

A deployment occurring before an incident is a useful investigative clue, but temporal proximity alone does not prove that the deployment caused the failure.

05 / Root-Cause Investigation

Distinguishing Symptoms from Underlying Causes

Microservices failures frequently propagate through dependency relationships. The component reporting an error may be several service calls away from the original source of degradation.

Consider a checkout service reporting payment timeouts. The immediate symptom is a failed payment operation, but the underlying cause may involve resource pressure, a downstream database, network disruption or an incorrect traffic policy.

Observed Symptom Possible Cause Validation Evidence
HTTP 503 Errors Unavailable application backends or failed upstream communication Endpoint readiness, gateway logs and service metrics
High p99 Latency Slow downstream calls, contention or retry amplification Trace waterfall, dependency metrics and resource data
Pod Restarts Application crashes, memory exhaustion or failed probes Container termination reasons, pod events and previous logs
Connection Failures DNS, network policy, service identity or destination failure Network telemetry, mesh policy logs and connectivity checks
Queue Growth Consumer saturation, reduced processing capacity or slow dependencies Queue depth, consumer throughput and processing traces

A strong investigation develops competing hypotheses and tests each against available evidence.

Engineers should avoid making production changes based solely on a dashboard correlation or an automated explanation.

When a cause remains uncertain, teams may still apply a reversible mitigation to restore service while continuing the investigation.

06 / AIOps and Intelligent Analysis

Using Analytics to Reduce Operational Noise

Artificial Intelligence for IT Operations (AIOps) refers to the application of analytics, statistical methods and machine learning to operational data.

Its objective is to help engineers detect meaningful patterns, reduce alert noise and prioritise investigations across complex environments.

AD
Anomaly Detection

Identify unusual changes in latency, error rates, traffic patterns or resource utilisation relative to an established baseline.

EC
Event Correlation

Group related alerts, identify shared dependencies and reduce the number of separate notifications presented to operators.

RA
Response Assistance

Summarise incident evidence, suggest investigation paths and recommend relevant runbooks for engineer review.

AIOps can be particularly useful in environments where hundreds of services generate thousands of telemetry events.

However, automated analysis depends on data quality, accurate service metadata and representative operational baselines.

Models may produce false positives, miss unusual failure modes or generate plausible but incorrect causal explanations.

AIOps should support evidence-based engineering decisions rather than replace validation and operational accountability.

07 / Incident Automation and Remediation

Automating Response Without Increasing Risk

Automation can reduce the time required to respond to known failure conditions. Common examples include alert enrichment, incident ticket creation, diagnostic data collection and execution of approved runbooks.

More advanced automation may apply corrective actions, but production changes should be governed by explicit safety controls.

Controlled Incident Automation Workflow Human-Governed
01
Trigger Validated alert or incident condition
02
Collect Gather logs, metrics, traces and context
03
Approve Evaluate risk and authorise corrective action
04
Validate Check recovery and revert if necessary

Safety principle: Human approval is appropriate for high-impact changes. Narrowly scoped, low-risk actions may be pre-authorised when they have defined permissions, rate limits, safeguards and rollback procedures.

Automation should be designed to avoid repeated actions that make an incident worse. For example, repeatedly restarting an overloaded service may increase instability rather than restore availability.

Useful safeguards include execution limits, idempotent operations, audit logging, approval gates, rollback mechanisms and post-action health checks.

08 / Production Incident Case Study

Diagnosing a Cascading Microservices Failure

Consider an online checkout platform containing an API gateway, order service, payment service and shared database.

Shortly after a deployment, customers begin experiencing slow checkout transactions and intermittent failures.

Production Incident · Checkout Degradation
STEP 01 / DETECT Service-Level Degradation

Checkout error rates and p99 latency increase. An SLO-based alert identifies customer-facing service degradation.

STEP 02 / INVESTIGATE Correlate Dependencies

Distributed traces reveal slow payment calls. Mesh telemetry shows increasing retries, while database metrics reveal elevated connection utilisation.

STEP 03 / MITIGATE Controlled Recovery

Engineers validate the contributing condition, apply an approved mitigation and monitor checkout latency, errors and dependency recovery.

Engineering interpretation: The evidence suggests a dependency-related performance problem with possible retry amplification. A complete root-cause analysis must establish why the downstream dependency slowed, whether the deployment contributed and which corrective action restored normal behaviour.

The incident demonstrates why a single monitoring signal is rarely sufficient to explain a complex microservices failure.

Application metrics reveal user impact, traces reveal affected execution paths, service mesh telemetry exposes communication behaviour and infrastructure metrics provide resource context.

Together, these signals support a more precise investigation than any individual dashboard.

09 / Engineering Incident Runbook

A Repeatable Investigation Method

A practical runbook should guide engineers through the incident without assuming the root cause in advance.

Microservices Incident Investigation Checklist
01
Confirm User Impact

Identify affected transactions, services, regions and reliability objectives.

02
Establish the Timeline

Compare the first symptoms with deployment events, configuration changes and infrastructure activity.

03
Inspect Service-Level Metrics

Review latency percentiles, error rates, traffic volume, saturation and error-budget consumption.

04
Follow Distributed Traces

Identify slow operations, failed spans, dependency delays and request critical paths.

05
Validate Infrastructure and Policies

Inspect Kubernetes workload health, resource pressure, network connectivity and service mesh policies.

06
Mitigate and Verify Recovery

Apply an authorised, proportionate action. Confirm recovery using independent service-level and dependency signals.

07
Document and Improve

Record evidence, contributing factors, recovery actions and improvements to monitoring, resilience and automation.

Post-incident reviews should focus on improving the reliability of the system and the effectiveness of operational processes.

Useful outcomes include better alerting, clearer dependency ownership, improved instrumentation, safer deployment controls and more reliable recovery procedures.

Network Insight · Engineering Principle

Observability is complete only when telemetry can support reliable operational decisions. Metrics detect degradation, logs provide event context, distributed traces reveal request execution, Kubernetes exposes workload state and dependency maps identify affected relationships. AIOps and automation can accelerate the response, but root-cause validation, change control and recovery verification remain essential engineering responsibilities.

Section 09 — Key Engineering Takeaways
✓ Incident response should follow a structured detection, triage, investigation, mitigation and learning process.
✓ Service-level indicators and SLO-based alerting help prioritise meaningful degradation.
✓ Correlating metrics, logs, traces and infrastructure events reduces investigative uncertainty.
✓ Event correlation does not establish causation; hypotheses require validation.
✓ AIOps can assist with anomaly detection, alert correlation and investigation prioritisation.
✓ Automated remediation requires appropriate safeguards, authorisation, rollback and verification.
✓ Post-incident learning should improve instrumentation, resilience and operational processes.
✓
Monitoring Microservices · Editorial Journey Complete Nine sections covering architecture, observability signals, distributed tracing, metrics, logging, OpenTelemetry, Kubernetes, service mesh, APM and incident response.
09 Final Engineering Investigation

Microservices Incident Command Centre

A production checkout service is failing. Investigate the incident using metrics, structured logs, distributed traces and dependency telemetry. Collect evidence, identify the root cause, evaluate AIOps recommendations and execute a safe remediation before producing your incident report.

AWAITING INCIDENT
01 / Incident Briefing
INC-2048 / Checkout Availability Degradation

The on-call SRE receives a high-priority alert. Checkout requests are failing, application latency has increased and the customer-facing error budget is being consumed. A recent deployment is visible in the change timeline, but the actual cause has not yet been confirmed.

SeveritySEV-2
EnvironmentProduction / Shop
Incident Clock00:00
Investigation StateNot started
02 / Live Incident Indicators
Checkout Success 99.9%
p99 Latency 310 ms
Error Rate 0.1%
Error Budget Used 0%
Evidence Collected 0/4
03 / Production Dependency Investigation

Service Dependency Map

APPLICATION TOPOLOGY

Select a service to inspect health, dependencies and incident context.

Service Inspector

APM DETAILS
Select a service to inspect its incident context.
04 / Correlated Observability Signals

Telemetry Investigation Console

METRICS / LOGS / TRACES
Investigation method: Correlate the time of the failure with changes, locate the failing dependency and use traces and logs to distinguish a symptom from a root cause.
05 / Evidence Collection & Root Cause

Collect Supporting Evidence

Review the telemetry before marking evidence. Four observations are available.

Root Cause Diagnosis

Which explanation best accounts for the incident?

Collect evidence and select a diagnosis.
06 / AIOps & Guarded Remediation

AIOps Recommendation Engine

ADVISORY ONLY
Start the incident and collect evidence to generate a simulated remediation recommendation.

Choose a Remediation

Rollback Inventory Deployment

Restore the previous known-good inventory application revision.

Restart All Microservices

Restart every application workload without correcting the configuration.

Increase Gateway Resources

Allocate additional CPU to the gateway without changing inventory.

Confirm the root cause before executing a remediation.
07 / Incident Event Timeline

Investigation & Response Events

INCIDENT JOURNAL
08 / Final Incident Report

Incident Response Scorecard

ENGINEERING ASSESSMENT
0/100
Telemetry Investigation 0 / 20
Evidence Collection 0 / 20
Root Cause Identification 0 / 25
AIOps Recommendation Review 0 / 10
Successful Remediation 0 / 25
Begin the investigation to generate your incident response assessment.
Engineering principle: Effective incident response connects detection, observability, dependency analysis, root-cause validation and controlled remediation. AIOps recommendations should remain explainable and subject to operational safeguards. This is a fictional training incident; all metrics, logs, traces, recommendations and remediation actions are simulated.
☕
Support Network Insight
Help support the development of interactive networking labs, BGP simulations, and educational content.
Support the Project
Matt Conran: The Visual Age
Latest posts by Matt Conran: The Visual Age (see all)

Comments are closed.