Runbook

The word comes from mainframe operations, where a run book listed the jobs, parameters and recovery steps operators needed to run a batch schedule. In modern operations a runbook is procedural and narrow: one alert or one routine task, with a precondition, a sequence of diagnostic and remedial steps, verification that the fix worked and an escalation point. The term playbook is often used interchangeably, but many organisations use playbook for the broader, scenario-level response (a ransomware playbook, a data breach playbook covering roles, communications and legal notification), with runbooks as the technical procedures invoked from it. The Google SRE book reports that recording best practices ahead of time in a playbook gives roughly a threefold improvement in mean time to repair compared with improvising.

A well-structured runbook for an alert states what the alert means and its user impact, links to the dashboards and queries needed to confirm it, lists ordered diagnostic commands with expected output, gives mitigations from least to most invasive (drain a node, roll back the last deployment, fail over a region), says when to stop and escalate and to whom, and records owner and last review date. Linking the runbook from the alert itself, for example through a runbook_url annotation on a Prometheus rule, means the responder reaches it in one step from the page.

Runbooks decay: infrastructure changes, commands reference hosts that no longer exist, and steps that were correct become dangerous. Common countermeasures are keeping runbooks in version control next to the service code, reviewing them after every incident that used them, exercising them in game days, and deleting alerts that have no meaningful runbook, since an alert with no documented action is usually one that should not page. Runbooks written as narrative prose rather than executable, checkable steps are harder to follow at 3 a.m.

The natural evolution is automation. Steps that are fully deterministic can be converted into scripts, then into runbook automation tools or executable notebooks, and finally into self-healing controllers that act without paging anyone, at which point the alert should be downgraded or removed. On the security side SOAR platforms implement the same idea for SOC workflows, executing enrichment and containment steps automatically when a SIEM alert arrives. NIST SP 800-61 Rev. 3 (April 2025) frames incident response within the NIST Cybersecurity Framework 2.0 functions and expects organisations to maintain documented response procedures; runbooks are the most concrete form of such procedures, and for entities under NIS2 or DORA they are part of the evidence that incident handling is actually operational.