Log management

NIST SP 800-92 (2006) defines log management as the process of generating, transmitting, storing, analysing and disposing of log data, and structures it as an infrastructure of generation, collection and storage, and analysis tiers. Its successor, SP 800-92 Rev. 1, "Cybersecurity Log Management Planning Guide", was released as an initial public draft in 2023 and reframes the topic as organisation-wide planning. CIS Controls v8 Control 8 (Audit Log Management) breaks it into safeguards, including establishing a process (8.1), collecting audit logs (8.2), ensuring adequate storage (8.3), standardising time synchronisation (8.4), centralising logs (8.9), retaining them for at least 90 days (8.10) and reviewing them (8.11). ISO/IEC 27001:2022 Annex A covers the same ground in controls 8.15 (Logging), 8.16 (Monitoring activities) and 8.17 (Clock synchronisation).

The technical pipeline has recognisable stages. Sources emit events via syslog (RFC 5424 for the message format, with RFC 5425 defining transport over TLS; the older BSD format is described in RFC 3164), Windows Event Log (collected by agents or Windows Event Forwarding), cloud audit APIs and application logs, increasingly as structured JSON. Collectors and forwarders buffer and ship events, preferably over authenticated, encrypted channels with back-pressure so that bursts do not drop data. Parsing and normalisation map vendor fields to a common schema such as Elastic Common Schema (ECS) or the Open Cybersecurity Schema Framework (OCSF); enrichment adds asset, identity and geolocation context. Storage is usually tiered into hot, warm and cold or archive layers with different query performance and cost.

Time is a first-class concern. Every source should synchronise to a common reference via NTP (RFC 5905) or PTP, logs should record timestamps with time zone or in UTC, and the pipeline should keep both the event time and the ingestion time, because clock skew and delayed forwarding otherwise break correlation and timeline reconstruction. Integrity controls include forwarding logs off the originating host quickly, storing them in a separate security account or tenant that administrators of the monitored systems cannot modify, write-once (WORM) or object-lock storage, and hash chaining or signing for evidential use.

Common failure modes are silent ones: a source stops sending after an agent update or certificate expiry, a parser change drops a field that detection rules depend on, or volume-based licensing leads teams to exclude high-value but noisy sources such as DNS or process creation. Health monitoring of log sources (last-seen time per source, expected volume ranges) is therefore part of log management itself. Log management differs from a SIEM, which consumes the managed data to correlate and alert, and from observability platforms, which use logs, metrics and traces primarily for reliability rather than security and often keep them for much shorter periods.