Observability

The term comes from control theory, where Rudolf Kálmán defined in 1960 that a linear system is observable if its internal state can be reconstructed from its outputs over a finite time. Software engineering borrowed the word in the mid-2010s, notably through Twitter's observability team and later Honeycomb, to describe the ability to explain arbitrary system behaviour from emitted telemetry without shipping new code. The distinction from monitoring is that monitoring covers known unknowns with predefined checks, whereas observability targets unknown unknowns in distributed systems whose failure modes cannot all be anticipated.

The popular three pillars framing (logs, metrics, traces) describes data types rather than the property itself, and is criticised for encouraging three disconnected silos. What makes a system observable in practice is correlation and dimensionality: every event carries the same identifiers (trace ID, service.name, deployment version, tenant, region) so an investigator can pivot from a latency spike on a metric to exemplar traces to the log lines of one span. A related approach stores wide structured events, one record per unit of work with dozens or hundreds of fields, and derives metrics from them at query time, which preserves high-cardinality fields such as customer ID that a pre-aggregated metric store would reject.

OpenTelemetry, a CNCF project formed in 2019 from OpenTracing and OpenCensus, provides the vendor-neutral layer: language APIs and SDKs, semantic conventions for attribute names, the OTLP wire protocol and the Collector for receiving, processing and exporting telemetry. Its signals are traces, metrics, logs and baggage, with profiling being added as a further signal. Decoupling instrumentation from backend means the storage and analysis vendor can be changed without re-instrumenting code.

Observability has real costs and failure modes. Telemetry volume grows with traffic and cardinality, so sampling, retention tiers and attribute budgets are engineering decisions with trade-offs in fidelity. Telemetry frequently contains personal data (IP addresses, user IDs, query parameters), which brings it within GDPR obligations on data minimisation and retention. Instrumentation that exists but is not queryable during an incident, because nobody knows the schema or the tooling is too slow, delivers little. Service level objectives give observability a purpose by defining which user-facing behaviour matters, and the same correlated data supports security investigation and threat hunting.