Anomaly detection

Dorothy Denning's 1987 paper "An Intrusion-Detection Model", developed alongside SRI's IDES system, established the core idea: maintain statistical profiles of subjects (users, hosts, processes) acting on objects, and flag observations that deviate significantly from the profile. The classic contrast is with misuse or signature detection, which matches known-bad patterns. NIST SP 800-94 uses the terms anomaly-based and signature-based detection, and adds stateful protocol analysis as a third method. Chandola, Banerjee and Kumar's widely cited 2009 survey distinguishes point anomalies (a single outlying value), contextual anomalies (normal in one context, abnormal in another, such as a login at 03:00) and collective anomalies (a sequence that is abnormal only as a whole, such as slow periodic beaconing).

Techniques range from simple statistics to machine learning. Baselines using mean and standard deviation (z-scores), exponentially weighted moving averages, and seasonal models that account for weekday and hour are still the workhorses in SIEM platforms. Unsupervised methods include isolation forest (Liu, Ting and Zhou, 2008), which scores points by how few random splits isolate them, local outlier factor, one-class SVMs, clustering and autoencoders whose reconstruction error serves as the anomaly score. User and entity behaviour analytics (UEBA) products apply these per identity and per device and add peer-group comparison, so that a finance clerk is compared with other finance clerks rather than with the whole organisation.

The central difficulty is the base-rate fallacy, analysed for intrusion detection by Axelsson (2000). Because genuinely malicious events are extremely rare, even a detector with a very low false-positive rate produces far more false alarms than true ones. Sommer and Paxson (2010) added that network anomaly detection differs from classic ML domains: the cost of errors is high, the "normal" class is extremely diverse and changing, ground truth is scarce, and a flagged anomaly often comes without an explanation an analyst can act on. Baselines also drift as organisations change, and patient attackers can deliberately stay within normal ranges or shift the baseline gradually, a form of poisoning. A new employee, a quarter-end batch job or a migration will look anomalous without being malicious.

In practice, effective deployments narrow the question: they model specific, well-understood behaviours (authentication geography, data egress volume, process parent-child relationships, service account usage), keep a training period free of known incidents, document the baseline window and thresholds, and route anomalies as context or risk scores that raise the priority of related rule-based alerts rather than as stand-alone alarms. Anomaly detection complements detection rules rather than replacing them: rules give precise, explainable coverage of known techniques, while anomaly models offer a chance to catch the unknown at the price of lower precision.