Distributed tracing

A trace is a directed acyclic graph of spans sharing one trace ID. Each span records a name, a span ID, its parent span ID, start and end timestamps, a kind (in OpenTelemetry: SERVER, CLIENT, PRODUCER, CONSUMER or INTERNAL), a status, key-value attributes, timestamped events such as exceptions, and optional links to spans in other traces, which is how batch and fan-in messaging work is modelled when one consumer span has many causal parents. The span model descends from Google's Dapper paper (2010), which inspired Zipkin, Jaeger, OpenTracing and OpenCensus; the last two merged into OpenTelemetry in 2019, now the de facto instrumentation standard.

Context propagation is the part that makes tracing distributed. Across HTTP the dominant format is the W3C Trace Context Recommendation: a traceparent header of the form version-traceid-parentid-flags, where the trace ID is 16 bytes (32 hex characters), the parent ID 8 bytes (16 hex), and the lowest bit of the flags byte marks the trace as sampled; all-zero IDs are invalid. A companion tracestate header carries up to 32 vendor-specific list members. Older systems use Zipkin's B3 headers, and messaging systems must carry the same context in message headers. Propagation breaks silently at any hop that does not forward headers, such as a proxy, a thread pool that loses in-process context, or a queue consumer without instrumentation, leaving orphaned fragments rather than an error.

Tracing every request is usually too expensive, so sampling is central. Head-based sampling decides at the root, typically probabilistically on the trace ID so every service reaches the same decision, and then propagates it through the sampled flag; it is cheap but cannot favour slow or failed requests because it decides before they happen. Tail-based sampling buffers all spans of a trace, for example in an OpenTelemetry Collector tier, and decides after completion, keeping errors and latency outliers at the cost of memory and a routing layer that sends all spans of one trace to the same instance.

Instrumentation is either automatic (agents or libraries that wrap HTTP servers, clients, database drivers) or manual spans around business logic, and attributes should follow OpenTelemetry semantic conventions so backends can interpret them. Traces differ from logs and metrics in that they encode causality and timing across process boundaries; exemplars link a metric data point to a representative trace ID, and injecting trace IDs into log records lets logs be joined to traces. Attributes can leak personal data or secrets such as full URLs with tokens, so span processors that redact attributes are part of a sound deployment.