Data poisoning

NIST AI 100-2 distinguishes poisoning by attacker goal: availability poisoning degrades the model broadly, targeted poisoning changes predictions on specific inputs, and backdoor poisoning implants a trigger that activates attacker-chosen behaviour while leaving clean accuracy intact. It also distinguishes by capability: control of labels, of data points, of the training procedure, or of the model itself (model poisoning, common in federated learning where clients submit updates). OWASP groups the LLM variants as LLM04:2025 Data and Model Poisoning, spanning pre-training corpora, fine-tuning sets, RLHF preference data and embeddings used for retrieval.

The classic backdoor is BadNets (Gu et al., 2017): stamp a small pixel pattern on a fraction of training images and relabel them to a target class; the trained network behaves normally until the pattern appears. Clean-label attacks (Shafahi et al., 2018, "Poison Frogs") avoid mislabelled samples altogether by crafting correctly labelled points whose features collide with a target, which defeats human label review. For web-scale data, Carlini et al. (2023) showed two practical routes: split-view poisoning, buying expired domains referenced in URL-list datasets such as LAION so that later downloads fetch attacker content (they estimated about USD 60 to control 0.01 % of LAION-400M), and front-running, timing malicious edits just before a Wikipedia snapshot. A 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that around 250 poisoned documents sufficed to implant a denial-of-service backdoor in LLMs from 600M to 13B parameters, suggesting the required number of samples stays roughly constant rather than scaling with dataset size.

Poisoning is hard to detect after the fact. Backdoored models pass standard benchmarks, and "sleeper agent" experiments (Hubinger et al., 2024) showed that supervised fine-tuning, RLHF and adversarial training did not reliably remove a conditional backdoor. Detection techniques - spectral signatures and activation clustering on training data, trigger reconstruction such as Neural Cleanse, loss-based outlier filtering - work for known trigger styles but offer no general guarantee.

The practical defence is therefore data governance: record provenance and licence for every dataset, pin snapshots by cryptographic hash rather than re-downloading URL lists, deduplicate and filter, limit and review contributions to fine-tuning and feedback data, keep holdout evaluation sets the data pipeline cannot touch, and red-team for trigger behaviour before release. The EU AI Act Art. 10 requires data governance for high-risk training data, and Art. 15(5) names data poisoning and model poisoning among the attacks high-risk systems must resist. Poisoning differs from adversarial examples, which fool a finished model at inference time, and from prompt injection, which manipulates context at run time; RAG poisoning - planting documents that a retriever will surface - sits between the two because it alters the data the model sees without retraining it.