The request touched fifteen services. One of them took four seconds. Which one — and what did it call, and what did that call, and where did the latency actually come from? The instrumentation pattern that makes distributed request paths visible end-to-end — and the sampling strategies that make it affordable at scale.
It's 3 AM. The pager says checkout P99 is 3.4 seconds. You open dashboards. Payment-service P99: elevated. Fraud-service: elevated. Order-service: elevated. Gateway: elevated. Everything is elevated. Nothing is clearly the culprit. You look at logs. Each service is doing what it's supposed to do; the problem isn't inside any one service — the problem is between them, somewhere in the chain of calls that a specific 4-second checkout request took across your architecture. And no single log stream, no single dashboard, no single per-service tool tells you what that request's journey actually looked like. The tools you have are per-service. The problem is cross-service.
request_id — 100M+ lines to search, 40-minute exerciseEvery other tool is either per-service (logs, APM, health) or aggregate (metrics, error rates). Distributed tracing is the only tool that gives you one specific cross-service request in full detail. That's not a small addition — it's the entire observability question at the layer where microservices actually live: between the services, not inside them.
The reason this gap exists is a design decision that made sense when we had monoliths and doesn't scale to distributed systems. Logs and metrics were built for a world where "one process handles one request start to finish" — the process log and the process metrics were the request log and the request metrics, so nothing was lost. Break that request into 15 processes across 15 network hops, and each process now only sees its slice of the request. The slices are still there in the logs; they just aren't glued back together. The tooling was never built to glue them. Distributed tracing is the missing glue.
The core insight is small: propagate a shared identifier — a "trace ID" — through every service the request touches, and let every service emit a "span" record for its slice of the work, tagged with the trace ID and the parent span ID. When you aggregate all the spans with the same trace ID, you get the full picture back. The waterfall you see in Jaeger or Tempo is just that aggregation, visualized by start time and nesting depth. The engineering effort in the tracing ecosystem — OpenTelemetry SDKs, W3C Trace Context standardization, sampling algorithms, storage backends — is all downstream of that one insight. Section 02 turns the insight into a working model; section 04 lets you feel what the visualization actually reveals when things get bad.
The data model is deceptively small. A trace is one end-to-end request — it has a globally-unique trace_id, and by definition it contains every span that participated in serving that request. A span is one unit of work — usually one service handling one call — and it records the service name, operation name, start time, duration, status (ok/error), and a set of key-value attributes. Every span has a span_id and a parent_span_id — those two IDs form the parent-child structure that lets a bag of spans be reconstructed into a tree, showing exactly which service called which. The whole visualization you see in Jaeger, Tempo, or any other tracing backend is just this reconstruction, drawn as a waterfall or a tree.
trace_id. Querying "all spans with trace_id X" reconstructs the whole request. The identifier is generated once (usually at the gateway or client) and propagated everywhere.parent_span_id. The pair (span_id, parent_span_id) across all spans in a trace forms a tree — the calling structure. That's how the waterfall knows what nests inside what.The mechanism that makes the shared trace_id work across processes is context propagation. When service A makes an HTTP call to service B, it injects the trace context (trace ID + current span ID) into an outbound header — the W3C standard is traceparent, which looks like 00-7f4a9b2e...-b2837f...-01. Service B reads that header, parses it, uses the trace ID for any spans it creates, and sets its span's parent to the injected span ID. When service B calls service C, it re-injects the header with its own span ID as the parent. The trace context flows with the request through every hop, and the parent chain builds itself as it goes. The service mesh from M.34 automates the HTTP-header side of this — sidecars propagate traceparent without app-level effort. But the mesh can only propagate what the app already has: the app is responsible for reading the incoming context and attaching it to outbound calls, especially into message queues, background jobs, and async workflows the mesh doesn't touch.
Where propagation breaks is where the trace ends. Async boundaries — Kafka messages, SQS jobs, scheduled tasks, background workers — need explicit context-propagation code (the trace context has to be embedded in the message envelope, then extracted on the consumer side). Every ecosystem has a pattern for this (OpenTelemetry's Kafka and SQS instrumentations, RabbitMQ headers, etc.), but it's not automatic. The most common tracing gap in a mature system is at the async boundary: a synchronous checkout looks perfect in the trace, but the moment the order goes into a queue for processing, the trace ends. The consumer picks up the message, does 30 seconds of work, and has no idea it's part of the original checkout. From the trace's perspective, the request just... stopped. The trace ends where the propagation ends. This is the single most reliable "why is my tracing incomplete?" answer in production, and understanding it saves days of debugging.
Tracing has a scaling problem the previous modules didn't have. A busy system handles millions of requests per hour; each trace has dozens or hundreds of spans; each span has attributes and events. Storing every span for every request produces terabytes per day, and querying that storage becomes its own performance problem. So every production tracing setup makes a sampling choice — keep some traces in full, discard others entirely. The interesting question is which ones to keep, and the answer isn't obvious. Keeping a random 1% might satisfy compliance and general observability, but it also means the specific slow checkout your on-call needs to see has a 99% chance of being one of the discarded ones. Sampling strategy is where distributed tracing's operational maturity actually shows up.
Every trace, every span, stored in full. Nothing is discarded. Fine for low-volume services (small internal APIs, dev environments, systems with tens of thousands of requests per day). Rare in mature production systems — the storage and query costs are prohibitive at meaningful traffic.
Flip a coin (or hash the trace ID mod 100) at request start; keep 1% (or whatever rate you configured). If sampled, propagate a "keep this" bit through the whole trace so every service records its spans. If not, everyone drops the trace. Cheap — the decision is local, the sampled flag rides in the same traceparent header. Predictable — you know exactly how many traces you'll store.
Every service emits every span to a collector; the collector holds spans in memory until the trace completes, then decides whether to keep it. Keep-decisions favor interesting traces — errors, slow ones (P99+), specific endpoints, anything matching a rule. The specific slow checkout your on-call needs to see is nearly always kept, because "slow" is the exact filter. Costs: memory + more infrastructure (a collector holding traces for their duration + a decision engine).
Sample rate adjusts based on traffic — lower during peak, higher during quiet periods. Alternative implementation: rate-limit to N traces/sec per endpoint or per service. Bounds cost regardless of traffic spikes; useful for services with very uneven load. Usually combined with tail-based so the adaptive part determines the volume ceiling and tail-based picks the interesting subset within it.
The head-vs-tail choice has a specific pathology worth naming. Head-based sampling at 1% catches ~1% of interesting traces because it doesn't know which are interesting. When the on-call gets paged about the 3.4-second P99 outlier, the trace they want to look at has a 99% chance of not being in the storage — because at request start, this trace looked like any other. The math is bad in exactly the way that matters: the higher the value of a particular trace (rare error, extreme latency), the lower the odds you have it. Tail-based sampling flips this: the interesting traces are the ones you keep, exactly because they're interesting. The tradeoff is infrastructural — a tail-sampling collector (like Grafana Tempo's tail-sampling processor, or a Jaeger/OpenTelemetry collector configured with tail_sampling) has to hold traces in memory long enough to make the decision, which means running a real service that scales with peak traffic. For mature systems, that infrastructure investment is worth it; the fresh-out-of-the-box teams typically start with head-based at a low rate and migrate to tail-based within a year of hitting real production load.
A common compound strategy in 2025-era systems: head-based at a high rate (e.g., 10%) for baseline coverage, plus tail-based rules that capture 100% of errors and 100% of slow requests. The head-based portion gives you statistical visibility into the "normal" traffic (so dashboards and aggregate views work); the tail-based portion gives you every anomaly for root-cause work. This composite pattern is what Datadog, Honeycomb, and Grafana Tempo all support out of the box; it's also easy to implement with an OpenTelemetry Collector pipeline. The general principle: sampling should be biased toward the traces you'll actually query, and in an incident response context, those are the anomalies. The lab in §04 lets you see this directly — inject a slow trace, see how each sampling strategy handles it, watch the head-based strategy miss the exact trace you need while the tail-based one catches it every time.
Below: a synthetic e-commerce checkout traced across 8 services. api-gateway → order-service → [inventory, pricing, tax, payment] → payment fans out to [fraud, ledger]. Toggle scenarios — baseline (everything healthy), slow leaf (fraud-service takes 800ms extra), error propagation (ledger-service errors, propagates up), mixed (inventory slow + tax errors). Then toggle sampling strategy — no sampling (100%), head-based (1%), tail-based (interesting-only). Watch spans stretch, watch errors cascade up the call tree, watch which sampling strategy captures which traces. The waterfall makes the root cause visible in a way logs and metrics never can.
Distributed tracing done well feels magical. Done incompletely, it feels like most of a picture with confusing holes in it. The five patterns below are the failure modes that survive even after teams have "adopted tracing" — the residual gaps where traces end unexpectedly, where interesting requests get discarded, where timing looks wrong, or where the observability picture just... stops halfway through the story. Recognizing them early is the difference between "tracing works for us" and "tracing works for us except in the exact cases we need it most."
Context propagation works automatically for synchronous HTTP calls (the mesh does it for you), but stops the moment the code drops a job into Kafka, SQS, or a Redis queue. The consumer picks up a message with no trace context; it starts a completely new trace. The end-to-end story splits into two disconnected traces — the request that queued the job, and the worker that processed it. The fix: explicit context injection into message headers (OpenTelemetry has instrumentations for every major broker); explicit extraction on the consumer side. It's not automatic, but it's a solved pattern once you know to look for it.
Head-based sampling at 1% catches ~1% of the rare bug that just happened. Aggregate metrics tell you it happened; logs from each service tell you something happened; but the one trace that would connect the dots isn't in the storage. The fix: tail-based sampling for anomaly capture (§03) — even at aggressive base rates, ensure 100% of errors and slow traces are kept. Complements head-based baseline coverage; doesn't replace it.
Each service records span timestamps from its own local clock. If service B's clock is 40ms ahead of service A's, service B's spans appear to start before service A finished sending the request — which is causally impossible. The fix: NTP or PTP for approximate clock alignment (millisecond-scale is usually enough); tracing UIs that handle small clock skew visually (nudging misaligned spans); and, importantly, understanding that the parent-child linkage is the source of truth for causality, not the timestamps. Timestamps are useful for latency estimation; parent IDs are useful for structure. Don't confuse them.
Span attributes are cheap individually and expensive in aggregate. Adding a high-cardinality attribute (user_id, session_id, request URL with variables inline) to every span creates enormous index size in the tracing backend. Query performance degrades; storage costs balloon. The fix: keep high-cardinality attributes but avoid them being indexed by default; use tools like Honeycomb designed for high cardinality; or scrub/hash user identifiers when they're not needed for debugging. Understanding what your backend indexes is essential before instrumenting.
A service that isn't instrumented — usually a legacy system, a third-party library, or a critical dependency the team doesn't own — appears in traces only via its call boundaries. Two seconds of work vanishes into "black hole" between the incoming and outgoing spans. The fix: OpenTelemetry auto-instrumentation for common libraries (JVM, .NET, Node, Python) catches most of this without code changes. For legacy systems, manual instrumentation of critical paths; for third-party services, sometimes acceptance that "this is a leaf" is the best you can do.
The composite lesson: distributed tracing is more social than it looks. Getting a technically correct picture requires every service to propagate context, every message queue to carry it, every legacy dependency to be instrumented, and every team to have agreed on sampling policy. The infrastructure is easy; the coordination is hard. This is the specific reason mature organizations spin up an "observability team" whose job is to own the tracing platform end-to-end: SDK versions, propagator libraries, collector configuration, backend storage, and — most importantly — the developer education that keeps teams from breaking propagation with well-intentioned refactors. Tracing is a shared infrastructure whose correctness depends on every team playing along, and the platform team is the one making that easy enough that they do.
The real-world tools have consolidated significantly. OpenTelemetry won the instrumentation-standard war around 2022 — the merger of OpenTracing and OpenCensus produced a vendor-neutral SDK spec that every major backend now supports. Jaeger (originally Uber, CNCF) is the go-to open-source backend for self-hosted setups; strong query UI, good storage backends (Cassandra, Elasticsearch, Kafka), CNCF governance. Grafana Tempo is the newer contender — object-storage-native (S3, GCS), cost-effective at large scale, integrates natively with Grafana's stack. Datadog APM, New Relic, and Honeycomb are the leading commercial hosted options; Honeycomb is worth calling out separately for its high-cardinality-first design that makes ad-hoc trace exploration dramatically better than the alternatives. The choice is usually about ops preference and cost model rather than capability, because at this layer the capabilities have converged. New systems in 2025 near-universally instrument with OpenTelemetry SDKs and pick a backend based on team preference; the days of proprietary tracing SDKs are essentially over.
The terms every OpenTelemetry config, every Jaeger dashboard, every "why is this trace incomplete?" postmortem assumes you already know.
trace_id. Contains every span that participated in serving that request. The unit of analysis in distributed tracing.span_id, parent_span_id, start time, duration, status, and attributes. Spans nest via parent-child linkage to form the request's tree.traceparent header is the standard format.traceparent (version, trace ID, parent span ID, flags) and tracestate (vendor-specific extras). Format: 00-{trace_id}-{parent_id}-{flags}.Test the tracing intuition. Click an answer; explanation drops in instantly.
Perfect. The trace/span model, propagation, sampling tradeoffs, failure modes — all yours. Now: your services are distributed. Should their data be? M.37 is polyglot persistence.
The observability layer that binds the per-service slices back into one cross-service story.
Logs are per-service. Metrics are aggregate. Neither tells the cross-service story of a specific request. The trace does — a shared ID propagated across every hop, with a span per service, reconstructing causality end-to-end.
The mesh handles HTTP header propagation. But apps must propagate context across async boundaries — Kafka, SQS, background jobs. Every unpropagated boundary is a place the trace ends. The single most common tracing gap in production.
Head-based sampling optimizes for uniform coverage but misses the rare errors and P99 outliers that actually matter. Tail-based sampling keeps the interesting traces exactly because they're interesting. Compose them for the best of both.