Alerting

An alerting pipeline has two distinct stages that are often conflated. Rule evaluation periodically runs a query against a time-series store or log index and turns the result into alert instances; in Prometheus an alerting rule is a PromQL expression plus a "for:" duration, and an instance moves from inactive to pending to firing only when the expression has returned a non-empty result on every evaluation for that duration. Notification routing is a separate component, such as Alertmanager, which deduplicates instances, groups them by labels (group_by, group_wait, group_interval, repeat_interval), applies silences and inhibition rules (for example suppressing every per-service alert while a datacenter-wide alert fires), and delivers to receivers such as a paging service, chat or a ticket queue. Labels such as severity and team drive routing; annotations carry the summary, a dashboard link and usually a runbook_url.

The Google SRE book distinguishes three outputs of monitoring: alerts, where a human must act immediately; tickets, where a human must act but not now; and logging, which nobody needs to read unless investigating. Anything that pages but requires no action, or where the action could be scripted, is a defect in the alerting design. The same source recommends alerting on symptoms users feel (the four golden signals: latency, traffic, errors, saturation) rather than on causes such as high CPU, which may or may not hurt anyone. Cause-based alerts remain useful for predictable exhaustion, such as a disk projected to fill within hours, where predict_linear-style extrapolation gives lead time.

SLO-based alerting refines symptom alerting by paging on the rate at which the error budget is consumed. A burn rate of 1 spends exactly the budget over the SLO window; the SRE Workbook suggests, for a 99.9% SLO over 30 days, paging when the burn rate exceeds 14.4 over one hour (2% of the budget gone) or 6 over six hours (5%), and opening a ticket at burn rate 1 over three days (10%). Each condition is paired with a short window of about one twelfth of the long one, so the alert also resets quickly once the problem stops.

Common failure modes are alert fatigue from noisy thresholds, flapping around a threshold without hysteresis, missing alerts when the metric itself disappears (a series that stops being scraped yields no data rather than a breach, which is why absent() checks and a dead man's switch "watchdog" alert exist), and a monitoring system that shares fate with what it watches. Alerting also differs from SIEM correlation: SIEM detections target security events across heterogeneous logs, but both depend on the same discipline of tuning, ownership and documented response.