{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/data-poisoning","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/data-poisoning/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/data-poisoning/"},"term":{"en":"Data poisoning","da":"Dataforgiftning (data poisoning)"},"aka":{"en":["training data poisoning","model poisoning"],"da":["forgiftning af træningsdata"]},"domain":["ai","security"],"cluster":"ai-risk","layer":"training","status":"current","summary":{"en":"Slipping false or harmful examples into the data an AI learns from so that it later behaves the way an attacker wants.","da":"At smugle falske eller skadelige eksempler ind i de data, en AI lærer af, så den senere opfører sig, som en angriber ønsker."},"body":{"formal":{"en":"An attack on the integrity of machine learning in which an attacker adds, changes or wrongly labels examples in the training data so that the finished model makes chosen mistakes, often only when a secret trigger appears.","da":"Et angreb på integriteten i maskinlæring, hvor en angriber tilføjer, ændrer eller mærker eksempler i træningsdataene forkert, så den færdige model begår bestemte fejl, ofte kun når en hemmelig udløser dukker op."},"plain":{"en":"Like secretly swapping a few pages in a student's textbook - they study hard, pass most tests, but give the wrong answer exactly where the pages were changed.","da":"Som i al hemmelighed at bytte nogle sider i en elevs lærebog - eleven læser flittigt og klarer de fleste prøver, men svarer forkert præcis der, hvor siderne blev byttet."},"inPractice":{"en":"A pension fund trains a model to spot false claims partly on a public collection of examples; an attacker has planted hundreds of cases there marked “honest”, so claims with the same pattern later pass unchecked.","da":"En pensionskasse træner en model til at finde falske anmeldelser delvis på en offentlig samling eksempler; en angriber har plantet hundredvis af sager mærket “ærlig” i samlingen, så anmeldelser med samme mønster senere slipper igennem."},"whyItMatters":{"en":"The damage sits inside the model and stays hidden until the trigger appears, so organisations must know and control where every piece of their training data comes from.","da":"Skaden sidder inde i selve modellen og forbliver skjult, indtil udløseren dukker op, så organisationer skal kende og styre, hvor hver del af deres træningsdata kommer fra."}},"deepDive":{"en":"NIST AI 100-2 distinguishes poisoning by attacker goal: availability poisoning degrades the model broadly, targeted poisoning changes predictions on specific inputs, and backdoor poisoning implants a trigger that activates attacker-chosen behaviour while leaving clean accuracy intact. It also distinguishes by capability: control of labels, of data points, of the training procedure, or of the model itself (model poisoning, common in federated learning where clients submit updates). OWASP groups the LLM variants as LLM04:2025 Data and Model Poisoning, spanning pre-training corpora, fine-tuning sets, RLHF preference data and embeddings used for retrieval.\n\nThe classic backdoor is BadNets (Gu et al., 2017): stamp a small pixel pattern on a fraction of training images and relabel them to a target class; the trained network behaves normally until the pattern appears. Clean-label attacks (Shafahi et al., 2018, \"Poison Frogs\") avoid mislabelled samples altogether by crafting correctly labelled points whose features collide with a target, which defeats human label review. For web-scale data, Carlini et al. (2023) showed two practical routes: split-view poisoning, buying expired domains referenced in URL-list datasets such as LAION so that later downloads fetch attacker content (they estimated about USD 60 to control 0.01 % of LAION-400M), and front-running, timing malicious edits just before a Wikipedia snapshot. A 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that around 250 poisoned documents sufficed to implant a denial-of-service backdoor in LLMs from 600M to 13B parameters, suggesting the required number of samples stays roughly constant rather than scaling with dataset size.\n\nPoisoning is hard to detect after the fact. Backdoored models pass standard benchmarks, and \"sleeper agent\" experiments (Hubinger et al., 2024) showed that supervised fine-tuning, RLHF and adversarial training did not reliably remove a conditional backdoor. Detection techniques - spectral signatures and activation clustering on training data, trigger reconstruction such as Neural Cleanse, loss-based outlier filtering - work for known trigger styles but offer no general guarantee.\n\nThe practical defence is therefore data governance: record provenance and licence for every dataset, pin snapshots by cryptographic hash rather than re-downloading URL lists, deduplicate and filter, limit and review contributions to fine-tuning and feedback data, keep holdout evaluation sets the data pipeline cannot touch, and red-team for trigger behaviour before release. The EU AI Act Art. 10 requires data governance for high-risk training data, and Art. 15(5) names data poisoning and model poisoning among the attacks high-risk systems must resist. Poisoning differs from adversarial examples, which fool a finished model at inference time, and from prompt injection, which manipulates context at run time; RAG poisoning - planting documents that a retriever will surface - sits between the two because it alters the data the model sees without retraining it.","da":"NIST AI 100-2 skelner mellem forgiftningsangreb efter angriberens mål: availability-forgiftning forringer modellen bredt, målrettet forgiftning ændrer forudsigelser på bestemte input, og bagdørsforgiftning planter en udløser, der aktiverer adfærd valgt af angriberen, mens nøjagtigheden på rene data er uændret. Den skelner også efter angriberens muligheder: kontrol over labels, over datapunkter, over træningsproceduren eller over selve modellen (modelforgiftning, som er udbredt i federated learning, hvor klienter indsender opdateringer). OWASP samler LLM-varianterne som LLM04:2025 Data and Model Poisoning, der dækker fortræningskorpora, finjusteringssæt, præferencedata til RLHF og embeddings til retrieval.\n\nDen klassiske bagdør er BadNets (Gu et al., 2017): man stempler et lille pixelmønster på en andel af træningsbillederne og ændrer deres label til en målklasse; det trænede netværk opfører sig normalt, indtil mønsteret dukker op. Clean-label-angreb (Shafahi et al., 2018, \"Poison Frogs\") undgår helt forkert mærkede eksempler ved at konstruere korrekt mærkede datapunkter, hvis features kolliderer med et mål, så manuel gennemgang af labels ikke hjælper. For data i internetskala viste Carlini et al. (2023) to praktiske veje: split-view-forgiftning, hvor man køber udløbne domæner, som URL-baserede datasæt som LAION henviser til, så senere downloads henter angriberens indhold (de anslog omkring 60 USD for at kontrollere 0,01 % af LAION-400M), og front-running, hvor ondsindede redigeringer times lige før et Wikipedia-snapshot. En undersøgelse fra 2025 fra Anthropic, UK AI Security Institute og Alan Turing Institute fandt, at omkring 250 forgiftede dokumenter var nok til at plante en denial-of-service-bagdør i LLM'er fra 600M til 13B parametre, hvilket tyder på, at det nødvendige antal eksempler er nogenlunde konstant i stedet for at vokse med datasættets størrelse.\n\nForgiftning er svær at opdage bagefter. Modeller med bagdør består standard-benchmarks, og \"sleeper agent\"-forsøg (Hubinger et al., 2024) viste, at supervised fine-tuning, RLHF og adversarial training ikke pålideligt fjernede en betinget bagdør. Detektionsteknikker - spectral signatures og activation clustering på træningsdata, rekonstruktion af udløsere som Neural Cleanse, filtrering af outliers ud fra tab - virker på kendte typer udløsere, men giver ingen generel garanti.\n\nDet praktiske forsvar er derfor datastyring: registrér oprindelse og licens for hvert datasæt, lås snapshots med kryptografisk hash i stedet for at downloade URL-lister igen, dedupliker og filtrér, begræns og gennemgå bidrag til finjusterings- og feedbackdata, hold evalueringssæt uden for datapipelinens rækkevidde, og red team for udløseradfærd før udgivelse. AI-forordningens art. 10 kræver datastyring for træningsdata til højrisikosystemer, og art. 15, stk. 5, nævner data- og modelforgiftning blandt de angreb, højrisikosystemer skal kunne modstå. Forgiftning adskiller sig fra adversarielle eksempler, der narrer en færdig model under inferens, og fra prompt injection, der manipulerer konteksten under kørsel; RAG-forgiftning - at plante dokumenter, som en retriever vil finde frem - ligger midt imellem, fordi den ændrer de data, modellen ser, uden at gentræne den."},"edges":[{"type":"requires","to":"ai/model-training","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/prompt-injection","why":{"en":"Poisoning corrupts the model while it learns; prompt injection misleads a finished model while it is in use.","da":"Forgiftning ødelægger modellen, mens den lærer; prompt injection vildleder en færdig model, mens den bruges."},"confidence":"high","strength":"normal"},{"type":"exploits","to":"ai/training-data","why":{"en":"A model trusts whatever it learns from, so tampered examples become part of its behaviour.","da":"En model stoler på det, den lærer af, så manipulerede eksempler bliver en del af dens adfærd."},"confidence":"high","strength":"primary"},{"type":"causes","to":"ai/ai-bias","why":{"en":"Skewed or planted examples can push a model to treat some groups or cases unfairly.","da":"Skæve eller plantede eksempler kan få en model til at behandle visse grupper eller sager uretfærdigt."},"confidence":"medium","strength":"minor"}],"depth":2,"sources":[{"title":"NIST AI 100-2 - Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations","url":"https://doi.org/10.6028/NIST.AI.100-2e2025","tier":"standard","publisher":"NIST"},{"title":"OWASP Top 10 for LLM Applications - LLM04 Data and Model Poisoning","url":"https://genai.owasp.org/llmrisk/llm042025-data-and-model-poisoning/","tier":"reference","publisher":"OWASP"},{"title":"ENISA - Multilayer Framework for Good Cybersecurity Practices for AI","url":"https://www.enisa.europa.eu/publications/multilayer-framework-for-good-cybersecurity-practices-for-ai","tier":"reference","publisher":"ENISA"},{"title":"A small number of samples can poison LLMs of any size (Anthropic, UK AI Security Institute, Alan Turing Institute, 2025)","url":"https://www.anthropic.com/research/small-samples-poison","tier":"other","publisher":"Anthropic"}],"draft":true}