Monitoring

The Google SRE book frames monitoring as collecting, processing, aggregating and displaying real-time quantitative data about a system, and draws a key line between black-box and white-box monitoring. Black-box monitoring tests externally visible behaviour the way a user would, with synthetic HTTP requests, DNS lookups or scripted login journeys from several locations; it detects symptoms that are happening now but says nothing about causes or imminent failure. White-box monitoring relies on internals exposed by the system itself (metrics endpoints, logs, runtime statistics) and can reveal a queue that is filling or retries masking errors before users are hurt. Real user monitoring, which collects timings from actual browsers or apps, complements both.

Architecturally, monitoring has evolved from check-based systems to time-series systems. Nagios (released in 1999 as NetSaint) and its descendants run plugins that return OK, WARNING, CRITICAL or UNKNOWN per host and service, while SNMP polling and traps remain common for network gear and appliances. Prometheus, started at SoundCloud in 2012 and accepted into the CNCF in 2016 as its second project after Kubernetes, popularised pulling labelled metrics into a time-series database and expressing both dashboards and alert conditions as queries over it, with service discovery keeping the target list in step with dynamic infrastructure. Dashboards (Grafana being the common front end) serve humans; alert rules serve machines, and the two should not be confused with each other.

Frequent failure modes include monitoring the monitoring system from within the same failure domain, so a network or cluster outage takes out both the service and the system meant to report on it; stale targets after infrastructure changes, which silently stop producing data; thresholds copied from defaults rather than derived from user impact; and dashboards with hundreds of panels that nobody can interpret during an incident. A meta-monitoring check from an independent location and an always-firing watchdog alert address the first two.

Monitoring also has a compliance role. ISO/IEC 27001:2022 Annex A control 8.16, Monitoring activities, expects networks, systems and applications to be monitored for anomalous behaviour, and security teams typically consume the same telemetry through a SIEM. Compared with observability, monitoring is the narrower practice of watching predefined signals for known failure modes; observability is the system property that allows new questions to be asked during novel incidents. In practice the two share pipelines, and mature teams still rely on monitoring for alerting while using high-dimensional data for investigation.