{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/adversarial-example","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/adversarial-example/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/adversarial-example/"},"term":{"en":"Adversarial example","da":"Adversarielt eksempel"},"aka":{"en":["adversarial input"],"da":["adversarielt input"]},"domain":["ai"],"cluster":"ai-risk","layer":"inference","status":"current","era":2013,"summary":{"en":"An input altered on purpose, often in ways people cannot see, so that a trained AI model gives a confident but wrong answer.","da":"Et input, der er ændret med vilje, ofte usynligt for mennesker, så en færdigtrænet AI-model svarer forkert med stor sikkerhed."},"body":{"formal":{"en":"An input to a trained machine learning model, usually a neural network, that an attacker has shifted by a small, calculated amount so that the output changes to a wrong or chosen answer at inference time; the model itself is not modified.","da":"Et input til en trænet maskinlæringsmodel, typisk et neuralt netværk, som en angriber har forskudt en lille, beregnet smule, så resultatet under inferens skifter til et forkert eller bestemt svar; selve modellen ændres ikke."},"plain":{"en":"Like an optical illusion made for a machine - a few dots a person would never notice make the computer see a cat as a toaster.","da":"Som en optisk illusion lavet til en maskine - nogle få prikker, et menneske aldrig ville lægge mærke til, får computeren til at se en kat som en brødrister."},"inPractice":{"en":"A municipality's IT operations manager finds that its AI-based malware filter can be fooled - changing a few unused parts of a harmful file, without touching what it does, makes the filter mark it as safe.","da":"En kommunes IT-driftsansvarlige opdager, at det AI-baserede malware-filter kan narres - ændrer man nogle få dele af en skadelig fil, som filen ikke selv bruger, markerer filteret den som sikker, selvom den virker præcis som før."},"whyItMatters":{"en":"A model can pass every ordinary test and still fail against an opponent who shapes the input, so high accuracy says little about safety where AI filters malware, checks faces or steers vehicles.","da":"En model kan bestå alle almindelige test og alligevel fejle over for en modstander, der former inputtet, så høj træfsikkerhed siger lidt om sikkerheden, når AI filtrerer malware, genkender ansigter eller styrer køretøjer."}},"deepDive":{"en":"Adversarial examples were described by Szegedy et al. in \"Intriguing properties of neural networks\" (2013), who found that imperceptible perturbations could flip an ImageNet classifier's output and that the same perturbed images often fooled other networks trained on different data. Goodfellow, Shlens and Szegedy (2014) argued that the cause is not overfitting but excessive linearity in high-dimensional models: a tiny change in every input dimension adds up to a large change in the logit. Their Fast Gradient Sign Method computes x' = x + ε · sign(∇x J(θ, x, y)) in a single backward pass, where J is the training loss and ε bounds the L∞ norm of the perturbation.\n\nMost attacks are framed as a constrained optimisation: find δ with ‖δ‖p ≤ ε that maximises the loss (untargeted) or minimises the loss for a chosen label (targeted). Projected Gradient Descent (Madry et al., 2017) iterates FGSM-like steps and projects back onto the ε-ball; Carlini and Wagner (2017) optimise a margin-based objective under L2 or L0. The threat model matters: white-box attackers have gradients, black-box attackers either query the model (score- or decision-based attacks) or craft examples on a local surrogate and rely on transferability. Typical research budgets, such as ε = 8/255 under L∞ on CIFAR-10, measure robustness against small pixel noise, not against every change a real attacker might make.\n\nPhysical-world attacks show the problem survives printing and cameras: stickers on stop signs (Eykholt et al., 2018) and a 3D-printed turtle classified as a rifle (Athalye et al., 2018, using Expectation over Transformation). In security products the constraint is functional rather than perceptual - a malware sample must still execute, so attackers modify padding, appended bytes, unused sections or imports. For language models the analogue is an adversarial suffix of tokens, as in the GCG attack on aligned LLMs, which links this term to jailbreaks.\n\nNIST AI 100-2 classifies adversarial examples as evasion attacks at deployment time, as opposed to poisoning (training time) and privacy attacks. Defences have a poor track record: many published defences relied on gradient masking and were broken by adaptive attacks (Athalye, Carlini and Wagner, 2018, \"Obfuscated Gradients\"). Adversarial training with PGD remains the strongest empirical defence but costs several times the normal training compute and usually reduces clean accuracy; randomised smoothing (Cohen et al., 2019) gives certified L2 guarantees only for small radii. Robustness claims should therefore be stated as a threat model plus an attack budget and evaluated with adaptive attacks, with public benchmarks such as RobustBench as reference points.","da":"Adversarielle eksempler blev beskrevet af Szegedy et al. i \"Intriguing properties of neural networks\" (2013), som viste, at umærkelige forstyrrelser kunne vende en ImageNet-klassifikators svar, og at de samme manipulerede billeder ofte også narrede andre netværk trænet på andre data. Goodfellow, Shlens og Szegedy (2014) argumenterede for, at årsagen ikke er overfitting, men at højdimensionelle modeller opfører sig for lineært: en lille ændring i hver eneste inputdimension summerer til en stor ændring i logit-værdien. Deres Fast Gradient Sign Method beregner x' = x + ε · sign(∇x J(θ, x, y)) i ét enkelt backward pass, hvor J er træningstabet, og ε begrænser forstyrrelsens L∞-norm.\n\nDe fleste angreb formuleres som et optimeringsproblem med en begrænsning: find δ med ‖δ‖p ≤ ε, der maksimerer tabet (utargeteret) eller minimerer tabet for en valgt klasse (targeteret). Projected Gradient Descent (Madry et al., 2017) gentager FGSM-lignende skridt og projicerer tilbage på ε-kuglen; Carlini og Wagner (2017) optimerer en marginbaseret målfunktion under L2 eller L0. Trusselsmodellen er afgørende: en white-box-angriber har adgang til gradienter, mens en black-box-angriber enten sender forespørgsler til modellen (score- eller beslutningsbaserede angreb) eller laver eksemplerne på en lokal surrogatmodel og udnytter transferability. Typiske forskningsbudgetter som ε = 8/255 under L∞ på CIFAR-10 måler robusthed mod lille pixelstøj, ikke mod enhver ændring, en reel angriber kan finde på.\n\nAngreb i den fysiske verden viser, at problemet overlever print og kamera: klistermærker på stopskilte (Eykholt et al., 2018) og en 3D-printet skildpadde, der blev klassificeret som et gevær (Athalye et al., 2018, med Expectation over Transformation). I sikkerhedsprodukter er begrænsningen funktionel frem for visuel - malware skal stadig kunne køre, så angriberen ændrer padding, tilføjede bytes, ubrugte sektioner eller imports. For sprogmodeller er det tilsvarende et adversarielt suffiks af tokens som i GCG-angrebet på alignede LLM'er, hvilket forbinder begrebet med jailbreaks.\n\nNIST AI 100-2 klassificerer adversarielle eksempler som evasion-angreb i driftsfasen, i modsætning til forgiftning (træningsfasen) og privatlivsangreb. Forsvarene har en dårlig historik: mange publicerede forsvar byggede på gradient masking og blev brudt af adaptive angreb (Athalye, Carlini og Wagner, 2018, \"Obfuscated Gradients\"). Adversarial training med PGD er stadig det stærkeste empiriske forsvar, men koster flere gange den normale træningsberegning og sænker som regel nøjagtigheden på rene data; randomized smoothing (Cohen et al., 2019) giver kun certificerede L2-garantier for små radier. Påstande om robusthed bør derfor angives som en trusselsmodel plus et angrebsbudget og evalueres med adaptive angreb, med offentlige benchmarks som RobustBench som pejlemærke."},"edges":[{"type":"requires","to":"ai/machine-learning","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/inference","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/data-poisoning","why":{"en":"An adversarial example fools a finished model while it is in use; data poisoning corrupts the model while it learns.","da":"Et adversarielt eksempel narrer en færdig model, mens den bruges; dataforgiftning ødelægger modellen, mens den lærer."},"confidence":"high","strength":"primary"},{"type":"exploits","to":"ai/neural-network","why":{"en":"A network's answer can swing sharply after small, well-aimed changes to its input that a person would ignore.","da":"Et netværks svar kan slå kraftigt om efter små, velrettede ændringer af input, som et menneske ville overse."},"confidence":"high","strength":"primary"}],"depth":2,"sources":[{"title":"Explaining and Harnessing Adversarial Examples (Goodfellow, Shlens, Szegedy, 2014)","url":"https://arxiv.org/abs/1412.6572","tier":"reference","publisher":"arXiv"},{"title":"NIST AI 100-2 - Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations","url":"https://doi.org/10.6028/NIST.AI.100-2e2025","tier":"standard","publisher":"NIST"}],"draft":true}