Monitoring Microservices: Understanding the Distributed System
Modern applications are no longer confined to a single server or a single execution environment. A request may cross API gateways, independently deployed services, container platforms, databases and external dependencies before producing a response.
This distributed architecture introduces flexibility, scalability and independent deployment. However, it also creates a fundamental operational challenge: understanding what is happening when application behaviour becomes unpredictable.
A service may appear healthy while its downstream dependency is experiencing high latency. An application may return successful responses while its performance gradually deteriorates. A single failing dependency can trigger retries, queue growth and cascading delays across multiple services.
From Application Request to Operational Insight
A conceptual request journey across independently deployed microservices
Request rate, latency, errors and resource saturation.
Detailed events, exceptions, state changes and context.
Request paths, span relationships and execution timing.
Correlating telemetry across services helps engineers connect symptoms to dependencies, identify bottlenecks and investigate failures.
Identify abnormal latency, rising error rates, unusual traffic patterns and resource pressure.
Bring together metrics, logs and traces to understand the behaviour of a distributed request.
Follow service dependencies, identify critical paths and isolate likely sources of failure.
Microservices Observability Explorer
Understand how metrics, logs and distributed traces work together to reveal what is happening inside a distributed application.
Unified Observability
Observability becomes powerful when different telemetry signals are correlated rather than examined independently.
Microservices Monitoring: From Visibility to Understanding
Microservices change how applications are designed, deployed and operated. Instead of running application functionality inside a single tightly integrated system, individual services are distributed across containers, hosts, clusters and cloud environments.
This architecture provides teams with greater flexibility to develop and scale individual services. However, it also introduces new operational complexity. Services communicate over networks, depend on external systems and may be deployed independently by different engineering teams.
A single user transaction may involve authentication, application logic, payment processing, databases, message queues and external APIs. Understanding the health of any individual component does not necessarily explain the behaviour of the complete transaction.
The central challenge is the growing distance between the cause of a failure and its visible effect. A database slowdown may first appear as an API latency problem. A failing downstream service may cause upstream retries, increasing traffic and resource consumption across otherwise healthy services.
From Monolithic Applications to Distributed Services
Traditional monolithic applications generally package multiple functional components into a common deployment unit. While these applications can still be complex and distributed, engineers often have fewer independently deployed services and network boundaries to investigate.
Microservices separate application capabilities into independently deployable components. Each service may have its own runtime, scaling characteristics, logs, metrics and operational dependencies.
Multiple application capabilities operate within a common deployment boundary.
Independently deployed services communicate through APIs and network connections.
The operational difference is significant. In a monolithic application, engineers may investigate application logs, runtime performance and database activity within a relatively consolidated environment.
In a microservices architecture, the same investigation may require correlating telemetry from dozens or hundreds of services, container instances, orchestration components and network connections.
Why Microservices Are Difficult to Monitor
The characteristics that make microservices attractive also introduce new failure modes. Independent deployments, elastic scaling and distributed communication make the application more dynamic, but they also reduce the usefulness of monitoring components in isolation.
Obscured Failure Paths
The source of a failure may be several service calls away from the user-facing symptom. Dependencies can hide the original cause of degradation.
Independent Components
Services have different release cycles, scaling behaviour and operational states. One service can fail while others continue operating.
Decentralised Execution
Hundreds of service instances may run across multiple hosts and clusters. Requests must be correlated across changing infrastructure.
These challenges become particularly visible during partial failures. An application does not always fail completely. Instead, a small percentage of requests may experience high latency, timeouts or inconsistent results.
Traditional infrastructure checks may report that every server is reachable and that CPU utilization remains within acceptable limits, while customers continue experiencing degraded application performance.
How a Single Dependency Can Affect the Entire Application
Microservices are independently deployed, but they are not operationally independent. A request frequently depends on several downstream services completing their work within an acceptable time.
When one dependency slows down, upstream services may wait longer for responses. Retries can increase traffic, connection pools may become exhausted and queues may begin to grow. Without suitable timeouts, backpressure and resilience controls, a local problem can spread across the application.
Engineering observation: The API gateway may report increased request latency, even though the original problem is a database dependency several service boundaries away. This is an illustrative failure scenario, not a live telemetry measurement.
This illustrates why monitoring individual components is insufficient. Engineers need visibility into service relationships, request dependencies, execution timing and the propagation of failures across the distributed system.
Monitoring Tells Us What Happened. Observability Helps Explain Why.
Monitoring and observability are closely related, but they address different aspects of operating distributed applications.
Monitoring focuses on collecting and evaluating known indicators of system health. Engineers define measurements, dashboards and alerts that identify expected failure conditions or performance deviations.
Observability extends this capability by providing enough contextual telemetry to investigate system behaviour, including failure modes that were not anticipated when monitoring rules were created.
Traditional Monitoring
What is happening?Monitoring tracks known conditions using predefined metrics, health checks, thresholds and alerts.
- Is the service available?
- Has CPU utilization increased?
- Is the error rate above its threshold?
- Has response time exceeded its objective?
System Observability
Why is it happening?Observability connects telemetry and execution context to investigate unexpected system behaviour.
- Which dependency caused the delay?
- Which requests were affected?
- Where did the failure propagate?
- What changed before degradation began?
Monitoring remains essential. Alerts, dashboards and service health indicators provide the initial signals that something requires attention. Observability makes those signals more useful by enabling deeper investigation into their causes.
For example, a monitoring system may detect that the 99th-percentile request latency has increased. Distributed tracing can identify which downstream operation consumed the additional time, while structured logs may reveal a database timeout, connection error or unexpected application state.
White-Box and Black-Box Monitoring
Effective microservices monitoring combines external observations with telemetry collected from inside applications and infrastructure.
Black-Box Monitoring
Observes a system from the outside, using synthetic requests, endpoint checks and externally measured availability or latency. It helps determine whether the service behaves correctly from a user's perspective.
White-Box Monitoring
Uses internal instrumentation to expose application state, request processing, resource consumption, dependencies and execution details. It helps engineers investigate why the system behaves as it does.
Both approaches are necessary. Black-box monitoring validates the externally observable experience, while white-box monitoring provides internal evidence for diagnosing failures.
A service can pass internal health checks while failing to deliver acceptable end-user performance. Conversely, an external failure may be difficult to diagnose without internal metrics, logs and distributed traces.
Observe the Request, Not Just the Infrastructure
The most important unit of investigation is often the complete user request. Engineers need to understand the sequence of operations that contributed to the response, including service execution, network communication, database queries and external calls.
This requires connecting application-level telemetry with infrastructure and platform signals. Kubernetes pod restarts, network timeouts, service-mesh errors, database saturation and application exceptions may all contribute to the same incident.
A reliable observability strategy therefore extends beyond collecting large volumes of data. It requires meaningful instrumentation, consistent metadata, service dependency context and the ability to correlate signals across the application.
A healthy component does not guarantee a healthy application. Microservices reliability must be evaluated across the complete request path. The goal is to connect application symptoms to the services, dependencies and infrastructure conditions responsible for them.
Section 01 — Key Engineering Takeaways
Microservices Architecture Explorer
Investigate a distributed checkout platform. Select services to inspect their health, inject failures, execute requests and observe how dependency problems affect the wider application.
Select any service node to inspect its simulated metrics and dependencies. Animated links show the most recent request path. This is an educational model, not a live Kubernetes or APM connection.
Start with the healthy scenario, then inject a dependency failure to investigate how microservices propagate errors.
Metrics, Logs & Distributed Traces
Understanding a distributed application requires more than knowing whether its individual services are running. Engineers must observe performance trends, investigate individual events and reconstruct how requests move between services.
Three complementary telemetry signals form the foundation of modern microservices observability: metrics, logs and distributed traces. Each provides a different perspective on application behaviour, and each becomes more valuable when correlated with the others.
Metrics describe measurable system behaviour over time. Logs record individual events and application context. Distributed traces reconstruct operations across service boundaries, helping engineers understand request execution and dependency timing.
The objective is not simply to collect three independent datasets. It is to connect them so that an abnormal metric can lead to a specific request trace, which can then lead to the relevant application logs.
Three Perspectives on the Same Distributed System
A microservices application produces different types of operational information. These signals answer different engineering questions and are suited to different forms of analysis.
Aggregated numerical measurements reveal changes in traffic, latency, errors, resource utilization and saturation.
Structured event records capture errors, exceptions, state transitions and contextual application information.
Distributed spans reconstruct the request journey and expose timing, service dependencies and failures.
Shared service metadata, trace context and consistent instrumentation connect telemetry into a coherent investigation.
None of these signals is a complete substitute for the others. Metrics provide efficient aggregated visibility but often lack individual request detail. Logs can expose precise events but may not show the complete request path. Traces provide execution context but can be affected by sampling and instrumentation gaps.
Metrics: Quantifying Application Health
Metrics are numerical measurements recorded over time. They allow engineers to observe how system behaviour changes, compare current performance against a baseline and identify conditions that require investigation.
Common microservices metrics include request throughput, response latency, error rates, resource utilization, queue depth and connection-pool saturation.
Understanding Time-Series Metrics
A metric measurement generally combines a value with a timestamp and identifying attributes or labels.
For example, request latency can be analysed by service, endpoint, deployment or region, provided the metric instrumentation includes the appropriate dimensions.
- Request rate measures incoming demand.
- Error rate identifies failed operations.
- Latency percentiles reveal slow requests.
- Saturation indicates resource pressure.
Percentile latency measurements are particularly important. An average response time may appear acceptable even when a subset of users experiences severe delays. The 99th-percentile latency, commonly called p99, describes a threshold at or below which approximately 99% of measured requests fall within the selected measurement window.
Tools such as Prometheus collect and store time-series metrics, while Grafana and other visualization platforms help engineers build dashboards and investigate performance trends.
Metrics should also be designed with storage efficiency in mind. Adding labels creates additional time series, and high-cardinality dimensions such as unique user identifiers or individual request IDs can significantly increase collection and storage costs.
Logs: Detailed Evidence of Application Behaviour
Logs record individual events generated by applications, infrastructure components and supporting platforms. They provide contextual evidence about errors, authentication activity, configuration changes, service lifecycle events and unexpected application states.
In distributed environments, unstructured text messages are difficult to correlate across independently deployed services. Structured logging improves searchability by representing events using consistent fields and formats.
Structured Logging
A structured log record can contain a timestamp, severity, service name, operation, trace identifier and error details.
Consistent fields make it possible to search across services and connect events belonging to the same distributed request.
- Timestamp identifies when an event occurred.
- Service name identifies its source.
- Trace ID links events to a request.
- Severity distinguishes event importance.
- Error context supports investigation.
Centralized logging platforms collect events from multiple sources so engineers can search, filter and analyse them without accessing individual service instances.
Technologies such as Elasticsearch, Logstash, Fluent Bit, Fluentd, Grafana Loki and cloud-native logging services support different stages of log collection, processing, storage and analysis.
Effective logging also requires operational discipline. Sensitive data should not be unnecessarily recorded, retention policies should reflect business and security requirements, and log volumes must be controlled to avoid excessive cost and noise.
Distributed Traces: Reconstructing the Request Journey
Distributed tracing follows a request as it crosses service boundaries. Instead of viewing isolated application events, engineers can reconstruct the operations that contributed to a complete transaction.
A trace contains spans representing individual units of work. Each span can record the operation name, start time, duration, attributes, status and relationship to its parent span.
Trace context propagates between services, allowing spans generated by different components to be associated with the same request. Standards such as W3C Trace Context support this propagation using identifiers carried in headers such as traceparent.
Understanding a Distributed Trace
A request may pass through an API gateway, authentication service, order service and database.
Each instrumented operation contributes a span to the overall trace, revealing the timing and relationships between different parts of the transaction.
- Trace ID identifies the request journey.
- Span ID identifies an individual operation.
- Parent relationships describe execution structure.
- Duration highlights expensive operations.
Bars compare individual span durations. They are not a chronological waterfall or a complete end-to-end latency calculation.
Tracing is particularly valuable when failures are distributed across several components. A slow API response may originate from a downstream database, an external service or a network connection rather than the API gateway itself.
Trace visualization systems such as Jaeger, Zipkin and Grafana Tempo help engineers explore service dependencies, execution timing and request-level failures.
Choosing the Right Signal for the Investigation
Metrics, logs and traces are complementary. The appropriate starting point depends on the operational question being investigated.
| Capability | Metrics | Logs | Traces |
|---|---|---|---|
| Primary purpose | Measure trends | Record events | Reconstruct operations |
| Typical data | Numeric time series | Event records | Spans and relationships |
| Best for | Health and alerting | Error investigation | Latency and dependencies |
| Common context | Service labels | Trace IDs and metadata | Trace and span IDs |
| Limitation | Limited request detail | Fragmented without correlation | Sampling and missing spans |
Connecting the Signals During a Production Incident
Consider an e-commerce application where customers report slow checkout operations. A dashboard shows increasing latency, but the initial metric does not reveal which component is responsible.
The checkout endpoint's p99 latency rises from its normal baseline to 850 ms. Error rates also begin increasing.
Affected request traces show excessive time spent waiting for a downstream database operation.
Correlated application logs report database connection timeouts, supporting the suspected dependency issue.
This workflow demonstrates the value of correlation. Metrics identify the degradation, traces narrow the affected execution path, and logs provide contextual evidence.
However, telemetry correlation does not automatically prove causation. Engineers should validate their hypothesis against dependency health, configuration changes, infrastructure events and other available operational evidence.
Collect telemetry with an investigation question in mind. Metrics should reveal meaningful service health, logs should preserve useful context, and traces should reconstruct critical request paths. Correlation across these signals transforms isolated measurements into operational understanding.
Section 02 — Key Engineering Takeaways
Three Pillars of Observability
Investigate a distributed checkout application using three complementary telemetry signals. Select an incident, inspect metrics, search structured logs, follow distributed traces and identify the most likely fault.
Metrics — What Is Happening?
PILLAR 01The chart is illustrative. The dashed line represents a demonstration latency threshold, not a universal SLO.
Service Dependency Map
TOPOLOGYMetrics Investigation
PILLAR 01Service-level metrics show normal latency, throughput and error rates.
Structured Log Investigation
PILLAR 02Search by service, severity, trace ID or message. Logs contain contextual events associated with the selected incident.
Distributed Trace Waterfall
TRACE-1001Each bar represents a simulated span positioned within the end-to-end checkout request.
Identify the Most Likely Incident Cause
Examine the telemetry before choosing an answer. Different scenarios produce different evidence.
Distributed Tracing: Following the Critical Path
In a distributed application, a single user request can trigger operations across multiple services, databases and external dependencies. Distributed tracing makes this execution journey visible by connecting individual operations into a coherent end-to-end trace.
Traditional logs and metrics provide valuable operational evidence, but they do not always reveal the sequence of service calls responsible for a particular request. Engineers may know that an API is slow without understanding which downstream operation caused the delay.
Distributed tracing addresses this challenge by recording the relationships and timing between operations. It enables engineers to examine a request across service boundaries, identify expensive dependencies and investigate where failures originated.
The fundamental objective is to reconstruct the complete request journey, rather than investigating each service in isolation.
Following a Request Across Microservices
Consider an application where a customer submits an order. The request enters through an API gateway, passes through authentication, reaches the order service and eventually accesses a database.
Each component performs a different operation. Some calls may execute sequentially, while others can run concurrently. Distributed tracing records these operations as spans and connects them using shared trace context.
4bf92f3577b34da6a3ce929d0e0e4736
The trace identifier allows telemetry generated across the participating services to be associated with the same distributed request. Individual operations receive their own span identifiers, while parent relationships describe how the operations are connected.
This distinction is important. A trace represents the broader request execution, while a span represents a particular operation within that execution.
Understanding Traces, Spans and Context
Distributed tracing relies on a small set of core concepts. Together, these concepts allow operations performed by different services to be reconstructed into a coherent execution model.
Trace
A trace represents a distributed operation, typically associated with a request or transaction. It groups related spans under a shared trace ID.
Span
A span represents an individual unit of work. It can contain timing, operation names, attributes, events and status information.
Trace Context
Trace context carries identifiers and sampling information across process boundaries, allowing distributed spans to remain linked.
A span typically records when an operation began, when it ended and how it relates to other operations. Additional attributes may identify the service, endpoint, database operation or relevant execution environment.
Parent-child relationships form a hierarchy. A root span may represent the incoming API request, while child spans represent authentication, application processing and downstream database calls.
In real applications, the span hierarchy may contain branches representing concurrent operations. The structure therefore describes execution relationships rather than simply listing services in the order they appear on a network diagram.
How Trace Context Crosses Service Boundaries
A distributed trace is useful only when participating services can preserve and propagate the request context.
When a service receives an instrumented request, it can extract the incoming trace context, create a new span and propagate the relevant context to downstream services.
The W3C Trace Context standard defines interoperable HTTP headers, including traceparent and tracestate, to support trace propagation between different frameworks and platforms.
In this example, the trace ID identifies the distributed trace. The parent ID identifies the span associated with the propagated context, and the flags include the sampled bit.
As the request crosses service boundaries, the trace ID normally remains consistent, while each newly created span receives its own span ID. Outgoing context is updated to reference the appropriate span for the next downstream operation.
Context propagation can be disrupted by missing instrumentation, unsupported messaging integrations or application code that fails to forward the context. These gaps can result in disconnected traces or incomplete request visibility.
Reading a Distributed Trace Waterfall
A trace waterfall visualizes spans along a common time axis. Each horizontal bar represents an operation, showing when it started and how long it took.
This representation allows engineers to compare execution timing, identify overlapping operations and locate spans associated with excessive latency.
Engineering observation: The database span consumes 240 ms of the 400 ms request window and accounts for much of the order service's execution time. This is illustrative trace data.
A waterfall must be interpreted carefully. Nested spans overlap with their parents, so simply adding every span duration together does not produce the total request latency.
Engineers must also distinguish between elapsed time and exclusive execution time. A service span may include time spent waiting for a downstream dependency rather than performing local computation.
Finding the Operations That Determine Request Latency
The critical path is the chain of dependent operations that determines the minimum completion time of a request, given the observed execution dependencies and durations.
In sequential request flows, a slow downstream operation can directly increase total response time. In concurrent execution, the longest dependent branch may determine when the request can complete.
Establish the total request duration and the start of the transaction.
Compare operation timing and parent-child relationships.
Identify the dependent operation contributing most to completion time.
Correlate the trace with logs, metrics and dependency health.
The longest individual span is not automatically the critical path. Engineers must account for execution relationships, parallel work and the timing of dependent operations.
For example, two database queries may execute concurrently. Improving the faster query may have no effect on overall request latency if the slower query still determines when the application can return its response.
Instrumentation, Collection and Trace Analysis
Distributed tracing requires services to generate spans and propagate context. Instrumentation can be implemented manually within application code or automatically through supported libraries and frameworks.
OpenTelemetry provides vendor-neutral APIs, SDKs, instrumentation libraries and collection components for generating and exporting telemetry.
Traces are commonly exported to a collector or observability backend, where they can be processed, stored and visualized.
Technologies such as Jaeger, Zipkin and Grafana Tempo provide different approaches to storing and exploring distributed traces.
Tracing also introduces operational considerations. Sampling strategies influence which requests are retained, instrumentation overhead must be controlled, and sensitive information should not be unnecessarily included in span attributes or events.
Investigating a Slow Checkout Transaction
Consider an e-commerce application where customers experience intermittent delays during checkout. Infrastructure dashboards show no obvious host failure, but application latency has increased.
The request takes 1.2 seconds to complete, significantly exceeding its normal baseline.
The distributed trace reveals that the order service spends most of its time waiting for a database operation.
Application logs and database metrics indicate connection-pool exhaustion and increased query wait time.
This case illustrates why tracing is particularly valuable in distributed systems. The visible symptom appears at the user-facing application, while the likely bottleneck exists several service boundaries downstream.
Distributed tracing narrows the investigation by exposing the execution path, allowing engineers to focus on the dependencies most relevant to the incident.
Investigate latency through execution relationships, not isolated service timings. A distributed trace reveals how operations depend on one another, where time is consumed and which downstream calls influence the end-to-end request. Correlating traces with metrics and logs helps turn a suspected bottleneck into a validated diagnosis.
Section 03 — Key Engineering Takeaways
Distributed Trace Investigation Lab
Follow a checkout transaction across microservices. Inspect individual spans, replay request propagation, investigate dependency latency and identify the operations responsible for end-to-end response time.
Request Propagation Topology
SERVICE GRAPHThe checkout workflow performs payment before inventory. Each link represents a synchronous request dependency.
Span Inspector
SPAN DETAILSPayment Authorisation
OKPayment service authorises the checkout transaction.
Request Execution Timeline
TRACE-3001Critical Path Analysis
Payment is the slowest individual operation in the healthy checkout request.
Trace Context Propagation
A distributed trace connects operations using trace IDs and parent-child span relationships.
This lab uses shortened demonstration identifiers. Production W3C trace IDs and span IDs use fixed-length hexadecimal values.
Which Dependency Is Responsible?
Select the operation most responsible for the scenario's abnormal latency or failure. For the healthy baseline, select No Incident.
Investigation Event Timeline
EVENTSMetrics, Golden Signals & Cardinality
Metrics provide the quantitative foundation of microservices monitoring. They transform application behaviour into measurable time-series data, allowing engineers to identify degradation, establish performance baselines and evaluate whether services are meeting their operational objectives.
In a distributed environment, however, collecting every available measurement does not automatically improve visibility. Engineers must choose meaningful signals, understand how metrics are aggregated, and manage the storage and query costs associated with metric labels.
Effective metrics engineering connects application health with the user experience. Instead of focusing exclusively on CPU, memory and infrastructure availability, teams should measure request latency, traffic, errors and resource saturation across the services that deliver application functionality.
Establishing a Meaningful Monitoring Baseline
The four golden signals provide a practical starting point for monitoring user-facing services. They focus attention on the measurements most likely to reveal performance degradation and operational pressure.
The time required to serve a request, including the behaviour of slow and unsuccessful operations.
The amount of demand placed on the service, such as requests per second or transactions.
Failed requests, incorrect responses, timeouts and application-level failures.
The degree to which a resource approaches its practical capacity or operating limit.
These signals should be measured at meaningful service boundaries. For example, an API gateway can expose overall request traffic and response latency, while a database service can expose connection utilization, query duration and queue pressure.
A useful monitoring baseline captures normal behaviour across different traffic periods. Without a baseline, engineers may struggle to distinguish a genuine incident from expected variations in application demand.
In this example, the elevated p99 latency and error rate suggest that a subset of requests is experiencing degradation. The next step is to correlate the metrics with request traces and application logs to identify the affected services.
RED and USE: Different Perspectives on System Health
The golden signals provide a broad monitoring framework. Two complementary methodologies help engineers organise metrics around service behaviour and resource utilization.
Request-Oriented Monitoring
RED focuses on the behaviour of services that process requests. It is particularly useful for APIs, microservices and other transaction-oriented components.
- Rate: Requests processed per second.
- Errors: Failed requests or operations.
- Duration: Time required to process requests.
Resource-Oriented Monitoring
USE focuses on the operational condition of resources and infrastructure components. It helps identify resource bottlenecks and capacity pressure.
- Utilization: Resource usage relative to capacity.
- Saturation: Queuing or excess demand.
- Errors: Resource-level failures.
RED is useful for understanding the customer-facing service experience, while USE helps investigate the infrastructure resources supporting those services.
For example, RED metrics may reveal increasing API response duration, while USE metrics show that a database connection pool or storage device is approaching saturation.
Why Average Latency Can Hide Performance Problems
Average response time provides a useful summary, but it can hide severe delays affecting a smaller percentage of requests.
Percentiles describe the distribution of observed response times. The p50 value represents the median, while p95 and p99 highlight the slower portion of the request population.
Example: Request Latency Distribution
Bars are scaled relative to the largest displayed value. The figures are illustrative rather than live measurements.
A service with a p50 latency of 42 ms may appear responsive for most users, yet a p99 latency of 850 ms indicates that the slowest portion of requests experiences substantially higher delays.
Tail latency is especially important in distributed systems because a transaction may depend on several downstream operations. Slow dependencies can increase end-to-end response time, particularly when requests must wait for multiple services to complete.
SLIs, SLOs and Error Budgets
Monitoring becomes more useful when measurements are connected to clearly defined service objectives. Instead of alerting on arbitrary thresholds, engineering teams can evaluate whether a service is delivering an acceptable user experience.
A Service Level Indicator (SLI) is a quantitative measurement of service behaviour. A Service Level Objective (SLO) defines the desired performance of that indicator over a specified period.
Service Level Indicator
99.95%Example objective for the proportion of eligible requests completed successfully during a defined measurement window.
The indicator must define which requests count as eligible and which responses qualify as good.
Error Budget
0.05%A 99.95% success SLO permits 0.05% of eligible requests to fall outside the objective.
Error budgets help teams balance reliability and the pace of change. When a service consumes its budget rapidly, engineers may prioritise stability improvements over additional deployments.
For a request-based SLO, the error budget can be calculated from the total number of eligible requests during the measurement window. Time-based availability objectives use a different measurement model.
Collecting and Querying Time-Series Data
Prometheus is a widely used open-source monitoring system designed for collecting, storing and querying time-series metrics. It commonly uses a pull-based model in which the Prometheus server scrapes metrics endpoints.
Applications can expose metrics directly through client libraries, while exporters make measurements available from systems that do not expose Prometheus-compatible metrics natively.
Prometheus supports several metric types, including counters, gauges, histograms and summaries. Each type serves different measurement and aggregation requirements.
A counter increases as events occur, making it suitable for request totals and error counts. The PromQL rate() function estimates the average per-second increase over a selected range.
Histograms can capture latency distributions and support percentile estimation across aggregated observations. Metric design should reflect the operational questions engineers need to answer.
Why Metric Cardinality Matters at Scale
Labels add dimensions to metrics. They allow engineers to analyse measurements by service, endpoint, region, deployment or other relevant attributes.
However, every unique combination of metric name and label values creates a distinct time series. As label combinations grow, the number of stored series can increase dramatically.
Example: Cardinality Growth
Consider a single metric with three independently varying label dimensions. The following values illustrate the potential number of unique series.
This is an upper-bound illustration assuming every label combination exists. Actual cardinality depends on which combinations are emitted.
High-cardinality labels can increase memory consumption, storage requirements, query complexity and telemetry costs. This becomes particularly problematic when labels contain values that change for every request.
Unique request identifiers, timestamps and unrestricted user identifiers are generally poor choices for Prometheus metric labels because they can create an extremely large number of unique time series.
Such identifiers are usually better suited to structured logs or distributed traces, where they provide request-level correlation without creating a new metric series for every transaction.
Turning Metrics into Actionable Alerts
Metrics are commonly evaluated over aggregation windows. The selected window affects how quickly a monitoring system detects changes and how much short-term variation appears in the resulting measurements.
Short windows can reveal rapid degradation but may produce noisy alerts. Longer windows smooth temporary fluctuations but can delay detection of emerging problems.
Effective alerting balances these characteristics. Alerts should represent meaningful operational conditions rather than every minor deviation from normal metric behaviour.
SLO-based alerting can use error-budget burn rates to identify conditions that threaten reliability objectives. Multi-window alerting strategies can combine rapid detection with protection against transient metric noise.
Monitoring systems should also account for missing telemetry, service discovery changes, delayed collection and differences between scrape intervals and query evaluation windows.
Investigating Elevated Tail Latency
Consider an order-processing service that normally handles 1,250 requests per second. Its average latency remains relatively stable, but p99 latency begins increasing during periods of peak demand.
p99 latency rises to 850 ms while request traffic increases above its normal operating baseline.
Database connection-pool utilization approaches capacity, and query wait times increase.
Distributed traces show longer waits on database operations, supporting a connection-capacity bottleneck hypothesis.
This investigation demonstrates why monitoring should connect user-facing performance with resource-level measurements. Request latency reveals the impact, while saturation metrics help explain the underlying pressure.
Measure what affects the user, then correlate it with what constrains the system. Golden signals and RED metrics reveal service degradation, while USE metrics expose resource pressure. SLIs and SLOs establish reliability expectations, and careful label design keeps the monitoring system scalable.
Section 04 — Key Engineering Takeaways
Live Metrics & SLO Dashboard
Explore the four golden signals, RED metrics, latency percentiles and service-level objectives. Inject production incidents, inspect changing telemetry and investigate how service degradation consumes an error budget.
Tail response time
Requests per second
Unsuccessful requests
Illustrative capacity usage
Live Latency Telemetry
ROLLING SAMPLESSamples update while traffic is running. The chart represents simulated p99 latency rather than measurements from a real monitoring system.
Latency Distribution
PERCENTILESSLO Control Panel
RELIABILITY ENGINEERINGError Budget Remaining
Calculating error budget...
RED Metrics Explorer
SERVICE HEALTHWhat Is the Primary Operational Problem?
Select a traffic scenario, examine the four golden signals and choose the diagnosis that best matches the simulated telemetry.
Monitoring Event Timeline
EVENTSCentralised Logging: From Events to Evidence
Logs provide detailed evidence of application behaviour. They record errors, state changes, requests, warnings and operational events that help engineers understand what happened inside individual services.
In a monolithic application, engineers may be able to investigate a problem using a relatively small collection of application logs. In a microservices environment, however, relevant events can be distributed across many services, containers, hosts and clusters.
A single failed transaction may generate log events in an API gateway, authentication service, order service, database client and external dependency. Without centralised collection and consistent context, connecting those events becomes difficult.
Centralised logging transforms isolated application messages into searchable, correlated operational evidence. It enables engineers to investigate incidents across the complete distributed system rather than examining individual components independently.
Why Traditional Log Investigation Breaks Down
Microservices generate large volumes of operational events. These events may be produced by short-lived containers, independently deployed services and infrastructure components that scale dynamically with application demand.
Logs stored only on individual hosts or container filesystems can become difficult to access, especially when workloads are rescheduled or terminated.
Distributed Sources
Relevant events are spread across different services, containers, hosts and cloud environments. Engineers need a unified view.
Ephemeral Infrastructure
Containers and pods may restart or disappear. Local log files cannot always be relied upon for historical investigation.
Volume and Noise
High request rates can produce enormous log volumes. Poorly designed logging increases cost and obscures useful evidence.
Centralised logging addresses these challenges by collecting events from multiple sources, applying consistent processing and making the resulting records searchable through a common investigation interface.
How Logs Move Through a Collection Pipeline
A typical logging architecture contains application log producers, collection agents, processing components, storage systems and analysis interfaces.
Applications emit log events to standard output, files or supported logging interfaces. Collectors forward the events to a processing or storage platform, where they can be indexed, labelled, retained and queried.
From microservices to searchable operational evidence
Architecture note: This is a conceptual pipeline. Some platforms combine processing, storage and analysis, while others use separate services or buffering systems between collection stages.
In Kubernetes environments, logging agents may run as DaemonSets to collect container logs from cluster nodes. Other architectures use sidecars, application-level exporters or managed cloud collection services.
Reliable collection pipelines should also consider buffering, backpressure, delivery failures and retention policies. If a central logging destination becomes unavailable, an unbuffered pipeline may lose events during the outage.
Designing Logs for Search and Correlation
Free-form text logs can be useful for human inspection, but they are harder to query consistently across multiple independently developed services.
Structured logging represents events using defined fields, commonly encoded as JSON. This enables filtering by service, severity, operation, trace ID, deployment and other contextual attributes.
Recommended Log Context
Useful log fields depend on the application, but a consistent schema improves the ability to investigate distributed failures.
- Timestamp: When the event occurred.
- Severity: DEBUG, INFO, WARN or ERROR.
- Service: Which component generated the event.
- Trace ID: Which distributed request it belongs to.
- Operation: The action being performed.
- Error details: The failure or exceptional condition.
Structured logging is especially valuable when combined with distributed tracing. A trace identifier can link application events generated by multiple services during the same request.
However, structured logging is not a substitute for good instrumentation. The recorded fields must be accurate, consistent and useful for the investigations engineers need to perform.
Understanding the Operational Event Stream
Log severity levels help distinguish normal application activity from conditions that may require attention. Common levels include DEBUG, INFO, WARN and ERROR.
Severity alone does not determine incident priority. A warning may indicate an emerging capacity problem, while an error may represent a transient failure that the application recovered from successfully.
In this example, the API gateway records the final failure, but the order service logs provide evidence of a database connection problem.
Searching only for API gateway errors would reveal the symptom without necessarily identifying the likely downstream cause.
Connecting Application Events to Distributed Requests
A trace ID provides a useful correlation key between distributed traces and application logs.
When the same trace identifier is included in relevant log events, engineers can search across services and reconstruct the event sequence associated with a specific request.
Select an affected distributed trace or obtain a trace ID from an application error.
Filter centralised log records using the trace ID and inspect events across participating services.
Compare the event sequence with trace timing and metrics to identify the likely cause.
Correlation requires consistent instrumentation. Applications must preserve trace context and include the relevant identifiers in their logging configuration.
In asynchronous systems, correlation may also require message identifiers, workflow identifiers or other application-specific context. A single trace ID does not always represent an entire long-running business workflow.
Collection, Storage and Investigation Technologies
Centralised logging architectures can be built using open-source components, commercial observability platforms or managed cloud services.
Different technologies focus on different parts of the pipeline, including collection, processing, storage, indexing, querying and visualisation.
| Technology | Typical Role | Engineering Use |
|---|---|---|
| Fluent Bit | Lightweight log collection and processing | Forward container and host events |
| Fluentd | Log collection and routing | Transform and deliver log records |
| Logstash | Event processing pipeline | Parse, enrich and transform events |
| Elasticsearch | Search and analytics storage | Index and query log data |
| Kibana | Log search and visualisation | Explore events and build dashboards |
| Grafana Loki | Log aggregation and querying | Investigate logs using labels and queries |
| Managed Cloud Logging | Hosted collection, storage and analysis | Centralise cloud and application events |
A common Elastic-based architecture combines collection agents, Logstash or other processing tools, Elasticsearch storage and Kibana visualisation.
Grafana Loki takes a different approach by indexing labels rather than the full contents of every log message, which influences how engineers design queries and choose label dimensions.
Platform selection should consider query requirements, data volume, retention, operational complexity, integration with traces and metrics, and total ownership cost.
Managing Security, Retention and Logging Costs
Centralised logging can introduce substantial operational and security considerations. Large distributed applications may generate significant volumes of events, including records containing sensitive application or infrastructure information.
Logging strategies should therefore include deliberate controls for access, retention, data minimisation and event volume.
Data Protection
Avoid recording passwords, access tokens and unnecessary personal information. Apply appropriate redaction and access controls.
Retention Policies
Define retention periods based on operational needs, security requirements, compliance obligations and storage cost.
Volume Management
Use appropriate severity levels, filtering and sampling policies to reduce noise without losing essential investigation evidence.
Logging systems should also be monitored as production services. Collection failures, dropped events, storage saturation and delayed ingestion can create visibility gaps precisely when engineers need telemetry most.
Investigating an Order Processing Failure
Consider a microservices application where customers intermittently receive HTTP 503 responses when submitting orders.
The API gateway reports the failed requests, but its logs do not identify the underlying database condition.
The API gateway reports HTTP 503 responses for order submission requests. Error metrics confirm an increasing failure rate.
Engineers search the centralised logging platform for the affected request's trace ID across services.
The order service records connection acquisition delays and database timeout errors.
The important distinction is that the API gateway recorded the visible symptom, while the downstream service recorded evidence of the likely cause.
Centralised logging made it possible to connect these events without manually accessing individual containers or hosts.
Logs become most valuable when they preserve context across distributed execution. Structured fields, consistent timestamps, trace correlation and centralised collection allow engineers to move from isolated error messages to evidence-based investigation. Logging quality should be designed into the application, not added only after an incident.
Section 05 — Key Engineering Takeaways
Centralised Logging Investigation Console
Investigate a distributed checkout application through structured logs. Search telemetry, filter services and severity levels, inspect JSON records, correlate trace identifiers and reconstruct the sequence of events leading to a production incident.
Distributed Application Log Stream
0 EVENTSSelect any log record to inspect its complete structured fields. Search is case-insensitive and applies to message text, service, severity, trace ID and error code.
Structured JSON Inspector
EVENT DETAILSSelect a log event.
Log Correlation Engine
TRACE CONTEXTCorrelated Request Timeline
NO TRACE SELECTEDIdentify the Root Cause
Investigate the log stream, filter errors and warnings, inspect structured fields and identify the most likely cause of the selected incident.
Investigation Activity
AUDIT TRAILOpenTelemetry & Instrumentation
Observability begins inside the application. Before metrics can be queried, traces can be visualised or logs can be correlated, software must generate meaningful telemetry describing its behaviour.
In a microservices architecture, instrumentation provides visibility into requests, operations, dependencies and failures across independently deployed components.
OpenTelemetry provides a vendor-neutral collection of APIs, SDKs, instrumentation libraries, semantic conventions and tools for generating, collecting and exporting observability data.
Rather than requiring every application to integrate directly with a particular monitoring platform, OpenTelemetry enables engineers to use standardised instrumentation and telemetry pipelines that can connect to different observability backends.
How Applications Produce Observability Data
Instrumentation introduces measurement and diagnostic information into application execution. It may be added automatically through supported frameworks and libraries, or manually through application code.
The goal is not simply to collect more data. Instrumentation should expose the operations, dependencies and failure conditions that engineers need to understand.
Automatic Instrumentation
Supported libraries and agents can instrument common frameworks, HTTP clients, database calls and messaging systems without requiring extensive changes to application code.
Framework-Level VisibilityManual Instrumentation
Developers create custom spans, metrics and attributes to represent important business operations, application-specific workflows and internal processing stages.
Business ContextContext Propagation
Trace context is propagated between services so independently recorded operations can be associated with the same distributed request.
Cross-Service CorrelationAutomatic instrumentation is useful for establishing broad coverage quickly. Manual instrumentation adds the application context that generic libraries cannot infer, such as a checkout validation stage or an order approval decision.
Both approaches are complementary. A production observability strategy often combines automatic framework instrumentation with carefully chosen application-specific measurements.
From Application SDKs to Observability Backends
OpenTelemetry separates telemetry generation from the systems used to store and analyse the resulting data.
Applications use instrumentation and SDKs to create telemetry. The data can be exported directly to a compatible backend or sent through an OpenTelemetry Collector for additional processing and routing.
Receivers · Processors · Exporters
Architecture note: The Collector is optional in some deployments, and the destination systems may use different protocols or storage models. The diagram illustrates a common centralised collection pattern.
OpenTelemetry commonly uses the OpenTelemetry Protocol (OTLP) to transport telemetry between SDKs, Collectors and compatible receivers.
OTLP supports different transport options, including gRPC and HTTP. The protocol transports telemetry; it does not itself provide a long-term metrics, logs or traces storage system.
Building a Flexible Telemetry Processing Pipeline
The OpenTelemetry Collector is a configurable telemetry service that receives, processes and exports observability data.
It can run as a central gateway, as an agent close to applications, or in architectures combining both deployment patterns.
Receivers
Accept telemetry from applications and other systems through supported protocols or collection mechanisms.
Processors
Apply configured operations such as batching, memory protection, filtering or attribute modification.
Exporters
Send processed telemetry to downstream Collectors or compatible observability platforms and services.
Collector pipelines are configured by telemetry signal. Metrics, logs and traces can have different processing requirements and destination systems.
For example, a production pipeline may batch trace data before export, filter unnecessary log events and attach environment metadata to telemetry records.
This simplified example accepts OTLP trace data, batches it and forwards it to another OTLP-compatible endpoint.
Security consideration: The insecure transport setting shown here is appropriate only for a controlled demonstration environment. Production deployments should use appropriate TLS, authentication, network restrictions and secret management.
Preserving Request Identity Across Services
A distributed trace can span multiple independently deployed services. To reconstruct the request journey, each participating service needs sufficient context to associate its spans with the same trace.
The W3C Trace Context standard defines HTTP headers such as traceparent and tracestate for carrying trace context between components.
Example: Context Propagation
The trace ID identifies the overall trace. The parent span identifier and trace flags provide additional context used to connect spans and communicate sampling decisions.
Context propagation is particularly important when requests pass through API gateways, service meshes, message brokers or asynchronous workers.
If context is not propagated across a service boundary, the downstream operation may appear as an unrelated trace, reducing the completeness of the investigation.
Adding Business Context to Distributed Traces
Automatic instrumentation can record common operations such as HTTP requests and database queries. However, it may not understand the business meaning of those operations.
Manual instrumentation allows developers to create spans around important application logic and attach attributes that improve troubleshooting.
This example creates a span for an order-processing operation, records a business attribute and captures exceptions before closing the span.
A functioning deployment also requires an appropriate OpenTelemetry SDK, span processor, exporter and provider configuration. The API code alone does not guarantee that spans will be exported.
Engineers should avoid attaching sensitive data or unbounded high-cardinality values to spans without considering privacy, storage and analysis requirements.
Balancing Diagnostic Detail and Operational Cost
High-volume microservices can generate large amounts of trace data. Recording and exporting every span may be expensive or unnecessary for some environments.
Sampling strategies allow engineers to control the amount of trace telemetry retained while preserving useful diagnostic information.
Head-Based Sampling
A sampling decision is made near the beginning of a trace, often using a configured probability or parent sampling decision.
Benefit: Sampling can reduce telemetry processing and export volume early in the request lifecycle.
Tail-Based Sampling
A sampling decision is made after enough trace data has been collected to evaluate characteristics such as errors or latency.
Benefit: Sampling policies can retain traces associated with significant failures or slow operations.
Tail-based sampling requires additional buffering and processing capacity because the decision is delayed until sufficient trace information becomes available.
Sampling can also introduce incomplete visibility if decisions or context propagation are not coordinated across services. Sampling policies should therefore be tested against real investigation requirements.
Avoiding Common Observability Design Mistakes
Instrumentation quality matters more than the raw quantity of telemetry. Excessive instrumentation can create unnecessary operational overhead, while incomplete instrumentation leaves critical diagnostic gaps.
| Design Area | Potential Problem | Recommended Approach |
|---|---|---|
| Span Boundaries | Missing important business operations | Instrument meaningful application stages |
| Context Propagation | Disconnected distributed traces | Preserve trace context across service boundaries |
| Attributes | Unbounded values or sensitive information | Use deliberate, privacy-aware attributes |
| Sampling | Important incidents are not retained | Align sampling policies with diagnostic needs |
| Collector Reliability | Telemetry loss during pipeline failures | Monitor queues, memory, retries and export errors |
| Service Identity | Telemetry cannot be attributed correctly | Apply consistent resource attributes such as service.name |
Consistent service identity is especially important in environments where workloads are frequently redeployed or rescheduled.
OpenTelemetry resource attributes provide information about the entity producing telemetry, helping observability systems group data by service, deployment and infrastructure context.
Finding a Missing Segment in a Distributed Trace
Consider a checkout request that passes through an API gateway, order service and payment service. Engineers can see the gateway and order spans, but the payment operation is missing from the distributed trace.
The trace visualisation ends at the order service even though application logs confirm that payment processing occurred.
Engineers inspect the outbound payment request, inbound context extraction and service instrumentation.
After correcting the identified context issue, new test requests produce correlated payment spans.
A missing span does not necessarily mean the underlying operation did not occur. It may indicate a gap in instrumentation, sampling, context propagation or telemetry delivery.
This distinction is important when using traces to investigate complex distributed failures.
Observability is engineered at the point where applications generate meaningful telemetry. OpenTelemetry provides a vendor-neutral instrumentation framework, while context propagation, Collector pipelines and sampling policies determine how effectively that telemetry supports real investigations.
Section 06 — Key Engineering Takeaways
OpenTelemetry Pipeline Builder
Configure an end-to-end observability pipeline. Enable application instrumentation, control telemetry processing, simulate exporter failures and investigate how collector configuration affects the delivery of metrics, logs and traces.
Instrumentation & Collector Configuration
PIPELINE CONTROLSEnabled Telemetry Signals
Pipeline Throughput
SIGNAL DELIVERYCollector Configuration
ILLUSTRATIVE YAMLThis is a teaching-oriented configuration preview. Component availability and valid settings depend on the installed Collector distribution and version.
Collector Diagnostic Console
EVENT STREAMIdentify the Pipeline Condition
Examine the configuration, delivery statistics and diagnostic console. Select the diagnosis best supported by the active scenario.
Kubernetes & Cloud-Native Monitoring
Kubernetes introduces a dynamic execution environment in which application workloads can be scheduled, restarted, replaced and scaled across multiple infrastructure nodes. Effective monitoring must therefore connect application behaviour with the changing state of the orchestration platform.
In a traditional environment, an application might run on a relatively stable collection of servers. Engineers can often associate performance problems with known hosts and predictable infrastructure dependencies.
Kubernetes changes that operational model. Pods can be recreated, workloads can move between nodes, and the number of application replicas can increase or decrease in response to demand.
Cloud-native monitoring must observe both the application and the platform responsible for running it. A healthy Kubernetes cluster does not automatically imply a healthy application, and an application failure does not necessarily originate inside application code.
Monitoring Applications Across a Dynamic Cluster
A Kubernetes observability architecture typically collects information from application containers, pods, nodes, Kubernetes control-plane components and supporting infrastructure.
These signals may be gathered through application instrumentation, Kubernetes APIs, metrics endpoints, container log collectors and infrastructure monitoring agents.
Architecture note: The illustrated collection layer represents multiple possible components, not necessarily a single agent. Real deployments may combine Prometheus, OpenTelemetry Collectors, Kubernetes exporters and cloud-managed services.
Kubernetes metadata provides important context for telemetry. Labels such as namespace, workload identity, pod name and node name help engineers determine where a signal originated.
Because individual pod identities can change frequently, long-term monitoring should also aggregate telemetry by stable workload and service dimensions.
Connecting Application Health to Infrastructure State
Effective Kubernetes monitoring requires visibility across several operational layers. Each layer answers a different set of troubleshooting questions.
Application
Request latency, error rates, throughput, business operations, distributed traces and application exceptions.
Container & Pod
Restarts, readiness, resource consumption, container termination reasons and pod scheduling state.
Node & Cluster
CPU and memory pressure, node availability, storage health, scheduling capacity and orchestration events.
Network & Dependencies
Service connectivity, DNS resolution, network policies, ingress traffic and external service availability.
These layers are interconnected. A request latency increase may originate from application logic, but it may also result from CPU throttling, an overloaded dependency, storage delays or network disruption.
Engineers should correlate signals between layers rather than assuming that the component reporting an error is responsible for causing it.
Monitoring Workloads That Continuously Change
Kubernetes controllers manage the desired state of application workloads. If a pod becomes unavailable or a deployment changes, the platform may create replacement pods.
Monitoring must account for these transitions because the disappearance of a pod does not necessarily indicate that its workload has stopped serving requests.
Illustrative Pod Replacement Journey
This is a simplified workload lifecycle illustration rather than the complete Kubernetes pod phase state machine. Real workloads can experience additional states, retries and termination paths.
A critical operational distinction exists between pod phase, container state and application readiness.
A pod may be in the Running phase while its application is not ready to accept traffic. Similarly, a container may repeatedly restart without the underlying node becoming unavailable.
Monitoring should therefore include readiness conditions, restart counts, termination reasons and workload availability rather than relying only on pod phase.
Metrics That Reveal Workload and Cluster Problems
Kubernetes exposes several categories of operational signals. Infrastructure metrics help identify resource pressure, while application metrics reveal the effect of that pressure on user-facing services.
CPU & Memory
Monitor CPU utilisation, CPU throttling, memory working sets, memory limits and out-of-memory termination events.
Pod & Workload Health
Monitor desired versus available replicas, restart counts, readiness conditions and scheduling failures.
Network & Storage
Monitor network errors, packet drops where exposed, DNS failures, storage latency and volume availability.
A common monitoring architecture uses Prometheus-compatible metrics endpoints and exporters to collect infrastructure and application measurements.
Tools such as kube-state-metrics expose Kubernetes object-state information, while kubelet and container-related metrics provide additional resource and runtime visibility.
Infrastructure metrics should be interpreted alongside application service-level indicators. High CPU utilisation is not automatically an incident, but high CPU combined with increased p99 latency and request errors may indicate a meaningful performance problem.
Following Requests Through the Cluster Network
Kubernetes networking introduces several layers between a client and the application container. Depending on the deployment, requests may pass through an external load balancer, ingress controller or Gateway API implementation, Kubernetes Service and the cluster networking layer.
Example Application Traffic Journey
Network investigation: A failed request can result from routing configuration, endpoint availability, DNS resolution, network policy enforcement, backend readiness or application failure.
Kubernetes Services provide stable network abstractions for changing groups of backend pods. EndpointSlice objects describe the endpoints associated with Services.
When an application becomes unavailable, engineers should verify whether suitable ready endpoints exist before assuming that the problem is caused by the external network.
NetworkPolicy objects may also influence connectivity when the installed network plugin supports their enforcement.
For advanced troubleshooting, network-level telemetry can complement application traces by exposing connection attempts, policy decisions, retransmissions or packet drops where supported by the underlying infrastructure.
Combining CLI Investigation with Observability Data
Observability dashboards help engineers detect patterns, but Kubernetes command-line inspection remains useful for validating the current state of workloads.
The following commands provide examples of operational checks during a service investigation.
The pod name in these commands is illustrative and must be replaced with the actual workload identifier. The --previous option is useful when a previous container instance is available, but it does not provide an unlimited historical log archive.
Commands reveal current or recently retained Kubernetes state. Centralised metrics, logs and traces provide the historical context needed to compare conditions before, during and after an incident.
Integrating Kubernetes with Managed Observability
Kubernetes monitoring can be implemented through self-managed observability systems or managed cloud platforms.
Cloud providers offer monitoring, logging and tracing integrations for managed Kubernetes environments, although collection coverage, configuration and pricing vary.
| Platform / Tool | Typical Role | Operational Value |
|---|---|---|
| Prometheus | Metrics collection and querying | Application and infrastructure metrics |
| Grafana | Observability dashboards and exploration | Visualise and correlate operational signals |
| kube-state-metrics | Kubernetes object-state metrics | Workload availability and object conditions |
| OpenTelemetry | Instrumentation and telemetry pipelines | Standardise collection and export |
| Google Cloud Observability | Managed monitoring, logging and tracing | Integrate GKE and Google Cloud telemetry |
| Amazon CloudWatch | AWS monitoring and log analysis | Observe AWS and EKS environments |
| Azure Monitor | Azure monitoring and analytics | Observe Azure and AKS workloads |
For example, Google Kubernetes Engine can integrate with Google Cloud Monitoring and Cloud Logging, while Cloud Trace supports distributed tracing investigations.
Cloud network telemetry, such as VPC Flow Logs where supported and enabled, may provide additional information about traffic patterns and network behaviour. It should not be treated as a replacement for packet-level investigation or application tracing.
Engineers should also monitor the telemetry pipeline itself, including ingestion delays, dropped records, collection errors and exporter health.
Investigating a Kubernetes Application Failure
Consider an order-processing application deployed across multiple Kubernetes worker nodes. Customers begin experiencing intermittent HTTP 503 responses shortly after a deployment update.
The cluster nodes remain Ready, but application availability has deteriorated.
Request metrics reveal increasing HTTP 503 errors. Distributed traces identify the order service as the affected application path.
Kubernetes metrics show reduced ready replicas. Pod events reveal readiness probe failures following the deployment update.
Engineers compare the deployment configuration and application startup behaviour to identify why new pods fail readiness.
The important lesson is that Kubernetes infrastructure can remain operational while the application becomes unavailable.
Node readiness alone does not guarantee sufficient application capacity, correct Service endpoint selection or successful end-to-end request processing.
By correlating application metrics, traces, logs and Kubernetes state, engineers can distinguish orchestration problems from application and dependency failures.
Cloud-native observability requires correlation across application, orchestration, infrastructure and network layers. Kubernetes can maintain the desired workload state, but engineers must still verify that the application is ready, reachable and delivering the expected service-level behaviour. The most useful monitoring systems connect workload identity, infrastructure conditions and application outcomes into a unified investigation.
Section 07 — Key Engineering Takeaways
Kubernetes Monitoring Explorer
Investigate a cloud-native application deployed across three Kubernetes worker nodes. Inspect pod lifecycle states, resource utilisation, deployment readiness, simulated cluster events and kubectl diagnostics to identify production incidents.
Select any pod to inspect its lifecycle state, restart count, resource usage and diagnostic context. Select a worker node heading to inspect node resources.
Resource & Workload Inspector
POD DETAILSDeployment Readiness
REPLICASSimulated kubectl Investigation
READ-ONLY COMMANDSSelect a command to inspect the simulated cluster.
Commands operate on the lab's simulated state. No connection to a real Kubernetes cluster is made.
Cluster Events
CONTROL PLANE EVENTSIdentify the Root Cause
Inspect pod states, restart counts, resource metrics, deployment readiness and kubectl output. Select the diagnosis that best matches the evidence.
Service Mesh, APM & Dependency Mapping
In a distributed microservices environment, application performance depends on more than the behaviour of individual services. It also depends on the communication paths, traffic policies and downstream dependencies connecting those services.
A service may appear healthy when examined in isolation while requests passing through it experience high latency, repeated retries or failures in downstream systems.
Service mesh technologies, application performance monitoring platforms and service dependency maps provide complementary ways to understand these relationships.
The objective is to move from monitoring individual components to observing the behaviour of the complete application system.
Introducing a Dedicated Service Communication Layer
A service mesh provides infrastructure capabilities for managing and observing communication between application services. Depending on the implementation, these capabilities may include traffic routing, encryption, identity enforcement, retries, timeouts and telemetry generation.
In a traditional sidecar-based architecture, a proxy is deployed alongside each participating application workload. The proxy handles selected inbound and outbound service traffic.
A control plane distributes configuration and policy to the data-plane proxies, allowing traffic behaviour to be managed consistently across many services.
Architecture note: This diagram illustrates a sidecar deployment model. Some modern service mesh architectures use shared node-level components and optional specialised proxies instead of deploying a sidecar beside every application. The animation represents logical service communication, not a literal packet path through the diagram.
Examples of service mesh technologies include Istio and Linkerd, although their architectures, features and operational requirements differ.
Service mesh observability can provide request rates, response codes, latency measurements and information about traffic between participating workloads.
However, mesh telemetry does not automatically reveal every internal application operation. Database processing, business logic and asynchronous workflows may still require application instrumentation.
Traffic Management, Security and Observability
Service meshes are frequently introduced to provide consistent communication controls without requiring every application team to implement those controls independently.
Traffic Management
Configure routing behaviour, retries, timeouts, traffic splitting and other supported communication policies.
Service Security
Support authenticated service communication, mutual TLS and identity-based policy enforcement.
Traffic Observability
Generate network and application-protocol telemetry that helps engineers inspect communication between services.
Traffic management policies require careful design. Retries may improve resilience during transient failures, but uncontrolled retries can amplify load on an already overloaded dependency.
Similarly, a timeout configured at one service boundary should be considered alongside the overall request deadline and the timeouts of downstream calls.
Understanding Application Behaviour Beyond the Network
Application Performance Monitoring (APM) focuses on the performance and reliability of application transactions, operations and dependencies.
APM platforms commonly combine application instrumentation, transaction traces, service-level metrics and dependency information to help engineers understand where application time is spent.
For example, an order-processing transaction might include request validation, inventory lookup, payment authorisation and database updates.
Each operation can contribute to the observed response time. Instrumentation allows engineers to identify slow operations, error-producing dependencies and changes in transaction performance.
| Technology | Primary Visibility | Typical Limitation |
|---|---|---|
| Service Mesh | Inter-service communication, traffic policies and proxy-observed metrics | Limited visibility into application internals without instrumentation |
| APM | Transactions, application operations and dependency performance | Coverage depends on supported instrumentation and configuration |
| Distributed Tracing | Individual request journeys and span timing | Missing context or sampling can leave incomplete traces |
| Infrastructure Monitoring | CPU, memory, network, storage and platform health | Resource metrics alone cannot explain application behaviour |
These technologies should not be treated as mutually exclusive. APM may use distributed tracing internally, while service mesh telemetry can enrich dependency maps and application dashboards.
Visualising the Relationships Between Microservices
A service dependency map represents the communication relationships between application components. Nodes typically represent services, while edges represent observed requests or other dependency relationships.
Dependency maps help engineers understand which components communicate, identify shared dependencies and assess the potential impact of failures.
Interpretation: The payment service is marked as degraded, while the shared database remains a possible contributing dependency. A dependency map identifies relationships and symptoms; it does not establish root cause by itself.
A useful dependency map may enrich edges with request volume, error rates and latency measurements.
Engineers can then identify whether a degraded dependency is isolated to one service or affects multiple consumers.
Dependency maps also support change-impact analysis. Before modifying a shared service, teams can inspect which applications rely on it.
Identifying Which Dependency Contributes to Delay
An application transaction may involve multiple downstream operations. APM and distributed tracing help engineers determine which operations contribute most significantly to the observed response time.
Consider an illustrative checkout transaction with several sequential processing stages.
Example Sequential Operation Durations
Illustrative timing: If these operations are strictly sequential and non-overlapping, their combined duration is 800 ms. In real traces, parent-child spans and parallel operations often overlap, so span durations must not simply be added together. A trace waterfall and critical-path analysis are needed to attribute end-to-end latency correctly.
In this example, the payment operation represents the largest individual delay. However, the next investigation should determine whether that delay originates in payment processing, network communication, retries or another downstream dependency.
APM systems can help distinguish application execution time from waiting on external calls, provided the necessary instrumentation is available.
Observing Encryption, Identity and Traffic Policies
Service mesh technology can introduce security and reliability controls into service-to-service communication.
These controls are operationally important because policy changes may affect application connectivity even when application code remains unchanged.
Mutual TLS
Supported meshes can authenticate participating workloads and encrypt service communication. Certificate or identity problems may prevent connections from succeeding.
Authorisation Policies
Identity-aware policies can restrict which services communicate. Engineers should inspect policy decisions when requests are unexpectedly denied.
Retries & Timeouts
Reliability policies can influence request behaviour. Excessive retries or poorly chosen timeouts can increase latency and amplify failures.
Security and traffic telemetry should be correlated with application errors. A connection failure may originate from a policy decision, an identity problem or an unavailable destination.
Engineers should also distinguish proxy-level observations from application-level outcomes. A successful network connection does not guarantee that a business transaction completed successfully.
Building a Unified Investigation View
The greatest value emerges when service communication telemetry, application instrumentation and infrastructure monitoring are analysed together.
For example, a service mesh may report increasing retries between two services. APM may reveal that the destination service is spending more time waiting on a database.
Kubernetes metrics may then reveal resource pressure affecting the database client or application workload.
| Signal | Investigation Question | Potential Evidence |
|---|---|---|
| Mesh Metrics | Is communication between services degrading? | Error rates, retries, latency and traffic volume |
| APM | Which application operation is experiencing delay? | Transaction breakdown and operation timing |
| Distributed Traces | Where did a specific request spend time? | Span hierarchy, timing and exceptions |
| Kubernetes Metrics | Is the workload resource constrained? | CPU throttling, memory pressure and pod readiness |
| Application Logs | What error or event occurred internally? | Exceptions, timeouts and dependency failures |
Consistent service naming, environment metadata and trace context propagation improve the ability to correlate these signals.
Without consistent identifiers, the same workload may appear under different names across monitoring systems, making investigations unnecessarily difficult.
Investigating a Degraded Payment Dependency
Consider a microservices checkout platform where customers report slow payment confirmation. The API gateway remains available, and the order service reports no obvious infrastructure failure.
However, end-to-end transaction latency has increased significantly.
Service mesh metrics reveal increased latency and retries between the order service and payment service.
Distributed traces identify long payment operations. APM highlights increased waiting time associated with a downstream call.
Engineers inspect payment logs, database metrics, timeout settings and retry behaviour to establish the actual source of delay.
Dependency mapping is particularly valuable in this situation because it identifies the services affected by a degraded component.
A shared dependency may create correlated failures across several otherwise healthy application services.
Engineers should therefore investigate both the failing component and its upstream and downstream relationships.
Distributed application performance is a property of the entire dependency graph, not merely the health of individual services. Service mesh telemetry exposes communication behaviour, APM reveals application transaction performance, and dependency maps help engineers understand the relationships between components. Correlating these views enables more effective troubleshooting and change-impact analysis.
Section 08 — Key Engineering Takeaways
Service Mesh Dependency Explorer
Investigate how requests travel between microservices. Explore application performance monitoring, service dependencies, proxy telemetry, mTLS, retries and circuit breakers while diagnosing latency and failures across a distributed application.
Service Dependency Inspector
SELECT A SERVICEService Mesh Traffic Policies
PROXY CONTROLSRequest Critical Path
NO TRACERun a request trace to inspect service latency.
Proxy & Application Events
TELEMETRY STREAMIdentify the Service Mesh Condition
Examine service dependencies, request latency, failures, traffic policies and trace spans. Select the diagnosis supported by the evidence.
Incident Investigation, AIOps & Automation
Observability becomes operationally valuable when engineers can use telemetry to detect incidents, understand their impact, investigate contributing factors and restore reliable service.
Modern microservices environments generate enormous quantities of metrics, logs, traces, infrastructure events and dependency information. Collecting these signals is only the beginning of an effective incident-management process.
Engineers must determine which signals are meaningful, identify relationships between apparently separate failures and distinguish symptoms from underlying causes.
The objective of incident investigation is not simply to identify a failing service. It is to establish what changed, why the system behaved differently and how normal service can be restored safely.
From Detection to Continuous Improvement
A structured incident-response lifecycle reduces uncertainty during production failures. It establishes a repeatable approach for detecting problems, investigating evidence and validating recovery.
Operational note: Incident response is iterative. Teams may investigate while mitigation is underway and revisit earlier hypotheses when new evidence becomes available. Learning and prevention continue after immediate service recovery.
Detection should focus on meaningful service degradation rather than every individual infrastructure event.
For example, a temporary pod restart may not affect users if sufficient healthy replicas remain available. In contrast, a sustained increase in checkout failures represents a direct service-level problem.
Incident severity should reflect business impact, affected users, service criticality and operational risk.
Moving from Infrastructure Alerts to Service-Level Signals
Traditional monitoring systems often generate alerts from infrastructure thresholds such as CPU utilisation, memory consumption or network interface errors.
These measurements remain important, but they do not always describe the experience of application users.
Microservices incident detection should therefore incorporate service-level indicators and application behaviour.
Latency
Detect sustained increases in request duration, especially at important percentiles and user-facing transaction boundaries.
Error Rates
Identify failed requests, dependency timeouts, exceptions and unsuccessful business transactions.
Saturation
Detect resource pressure, queue growth, connection exhaustion and other capacity constraints.
Dependencies
Observe downstream availability, retry amplification and cascading failures.
Service Level Objectives (SLOs) provide a useful foundation for alerting. Instead of reacting to every isolated measurement, teams can monitor whether a service is consuming its permitted error budget too quickly.
Multi-window, multi-burn-rate alerting can help distinguish severe short-lived degradation from slower persistent problems.
Alert thresholds and evaluation windows should be selected according to the service's reliability objectives, traffic patterns and operational requirements.
Building a Unified Incident Investigation
A production incident may generate many apparently unrelated alerts. An overloaded dependency can produce timeouts, retries, queue growth, application exceptions and infrastructure warnings.
Without correlation, each symptom may appear to represent a separate problem.
Important distinction: Correlation identifies related events and narrows the investigation. It does not automatically establish causation. Root-cause conclusions should be validated against independent evidence.
Consistent timestamps, service identifiers, deployment metadata and trace context improve correlation accuracy.
A useful investigation view should also include recent changes, such as deployments, configuration updates, policy modifications and infrastructure maintenance.
Understanding the Sequence of Failure
A reliable incident investigation reconstructs what happened before, during and after service degradation.
The sequence of events can help engineers distinguish a triggering change from secondary symptoms.
Illustrative Production Incident Timeline
A new order-service application version begins rolling out across Kubernetes pods.
Checkout p99 latency rises while the application remains partially available.
Failed checkout requests increase, triggering a service-level reliability alert.
Traces reveal prolonged payment calls. Service mesh telemetry shows increased retries.
Engineers apply a reviewed corrective action and monitor recovery of the affected transaction.
Error rates and transaction latency return to expected operating ranges.
Timeline reconstruction should account for clock differences, telemetry ingestion delays and incomplete records.
A deployment occurring before an incident is a useful investigative clue, but temporal proximity alone does not prove that the deployment caused the failure.
Distinguishing Symptoms from Underlying Causes
Microservices failures frequently propagate through dependency relationships. The component reporting an error may be several service calls away from the original source of degradation.
Consider a checkout service reporting payment timeouts. The immediate symptom is a failed payment operation, but the underlying cause may involve resource pressure, a downstream database, network disruption or an incorrect traffic policy.
| Observed Symptom | Possible Cause | Validation Evidence |
|---|---|---|
| HTTP 503 Errors | Unavailable application backends or failed upstream communication | Endpoint readiness, gateway logs and service metrics |
| High p99 Latency | Slow downstream calls, contention or retry amplification | Trace waterfall, dependency metrics and resource data |
| Pod Restarts | Application crashes, memory exhaustion or failed probes | Container termination reasons, pod events and previous logs |
| Connection Failures | DNS, network policy, service identity or destination failure | Network telemetry, mesh policy logs and connectivity checks |
| Queue Growth | Consumer saturation, reduced processing capacity or slow dependencies | Queue depth, consumer throughput and processing traces |
A strong investigation develops competing hypotheses and tests each against available evidence.
Engineers should avoid making production changes based solely on a dashboard correlation or an automated explanation.
When a cause remains uncertain, teams may still apply a reversible mitigation to restore service while continuing the investigation.
Using Analytics to Reduce Operational Noise
Artificial Intelligence for IT Operations (AIOps) refers to the application of analytics, statistical methods and machine learning to operational data.
Its objective is to help engineers detect meaningful patterns, reduce alert noise and prioritise investigations across complex environments.
Anomaly Detection
Identify unusual changes in latency, error rates, traffic patterns or resource utilisation relative to an established baseline.
Event Correlation
Group related alerts, identify shared dependencies and reduce the number of separate notifications presented to operators.
Response Assistance
Summarise incident evidence, suggest investigation paths and recommend relevant runbooks for engineer review.
AIOps can be particularly useful in environments where hundreds of services generate thousands of telemetry events.
However, automated analysis depends on data quality, accurate service metadata and representative operational baselines.
Models may produce false positives, miss unusual failure modes or generate plausible but incorrect causal explanations.
AIOps should support evidence-based engineering decisions rather than replace validation and operational accountability.
Automating Response Without Increasing Risk
Automation can reduce the time required to respond to known failure conditions. Common examples include alert enrichment, incident ticket creation, diagnostic data collection and execution of approved runbooks.
More advanced automation may apply corrective actions, but production changes should be governed by explicit safety controls.
Safety principle: Human approval is appropriate for high-impact changes. Narrowly scoped, low-risk actions may be pre-authorised when they have defined permissions, rate limits, safeguards and rollback procedures.
Automation should be designed to avoid repeated actions that make an incident worse. For example, repeatedly restarting an overloaded service may increase instability rather than restore availability.
Useful safeguards include execution limits, idempotent operations, audit logging, approval gates, rollback mechanisms and post-action health checks.
Diagnosing a Cascading Microservices Failure
Consider an online checkout platform containing an API gateway, order service, payment service and shared database.
Shortly after a deployment, customers begin experiencing slow checkout transactions and intermittent failures.
Checkout error rates and p99 latency increase. An SLO-based alert identifies customer-facing service degradation.
Distributed traces reveal slow payment calls. Mesh telemetry shows increasing retries, while database metrics reveal elevated connection utilisation.
Engineers validate the contributing condition, apply an approved mitigation and monitor checkout latency, errors and dependency recovery.
The incident demonstrates why a single monitoring signal is rarely sufficient to explain a complex microservices failure.
Application metrics reveal user impact, traces reveal affected execution paths, service mesh telemetry exposes communication behaviour and infrastructure metrics provide resource context.
Together, these signals support a more precise investigation than any individual dashboard.
A Repeatable Investigation Method
A practical runbook should guide engineers through the incident without assuming the root cause in advance.
Microservices Incident Investigation Checklist
Identify affected transactions, services, regions and reliability objectives.
Compare the first symptoms with deployment events, configuration changes and infrastructure activity.
Review latency percentiles, error rates, traffic volume, saturation and error-budget consumption.
Identify slow operations, failed spans, dependency delays and request critical paths.
Inspect Kubernetes workload health, resource pressure, network connectivity and service mesh policies.
Apply an authorised, proportionate action. Confirm recovery using independent service-level and dependency signals.
Record evidence, contributing factors, recovery actions and improvements to monitoring, resilience and automation.
Post-incident reviews should focus on improving the reliability of the system and the effectiveness of operational processes.
Useful outcomes include better alerting, clearer dependency ownership, improved instrumentation, safer deployment controls and more reliable recovery procedures.
Observability is complete only when telemetry can support reliable operational decisions. Metrics detect degradation, logs provide event context, distributed traces reveal request execution, Kubernetes exposes workload state and dependency maps identify affected relationships. AIOps and automation can accelerate the response, but root-cause validation, change control and recovery verification remain essential engineering responsibilities.
Section 09 — Key Engineering Takeaways
Microservices Incident Command Centre
A production checkout service is failing. Investigate the incident using metrics, structured logs, distributed traces and dependency telemetry. Collect evidence, identify the root cause, evaluate AIOps recommendations and execute a safe remediation before producing your incident report.
The on-call SRE receives a high-priority alert. Checkout requests are failing, application latency has increased and the customer-facing error budget is being consumed. A recent deployment is visible in the change timeline, but the actual cause has not yet been confirmed.
Service Dependency Map
APPLICATION TOPOLOGYSelect a service to inspect health, dependencies and incident context.
Service Inspector
APM DETAILSTelemetry Investigation Console
METRICS / LOGS / TRACESCollect Supporting Evidence
Review the telemetry before marking evidence. Four observations are available.
Root Cause Diagnosis
Which explanation best accounts for the incident?
AIOps Recommendation Engine
ADVISORY ONLYChoose a Remediation
Restore the previous known-good inventory application revision.
Restart every application workload without correcting the configuration.
Allocate additional CPU to the gateway without changing inventory.
