Adversarial example

Adversarial examples were described by Szegedy et al. in "Intriguing properties of neural networks" (2013), who found that imperceptible perturbations could flip an ImageNet classifier's output and that the same perturbed images often fooled other networks trained on different data. Goodfellow, Shlens and Szegedy (2014) argued that the cause is not overfitting but excessive linearity in high-dimensional models: a tiny change in every input dimension adds up to a large change in the logit. Their Fast Gradient Sign Method computes x' = x + ε · sign(∇x J(θ, x, y)) in a single backward pass, where J is the training loss and ε bounds the L∞ norm of the perturbation.

Most attacks are framed as a constrained optimisation: find δ with ‖δ‖p ≤ ε that maximises the loss (untargeted) or minimises the loss for a chosen label (targeted). Projected Gradient Descent (Madry et al., 2017) iterates FGSM-like steps and projects back onto the ε-ball; Carlini and Wagner (2017) optimise a margin-based objective under L2 or L0. The threat model matters: white-box attackers have gradients, black-box attackers either query the model (score- or decision-based attacks) or craft examples on a local surrogate and rely on transferability. Typical research budgets, such as ε = 8/255 under L∞ on CIFAR-10, measure robustness against small pixel noise, not against every change a real attacker might make.

Physical-world attacks show the problem survives printing and cameras: stickers on stop signs (Eykholt et al., 2018) and a 3D-printed turtle classified as a rifle (Athalye et al., 2018, using Expectation over Transformation). In security products the constraint is functional rather than perceptual - a malware sample must still execute, so attackers modify padding, appended bytes, unused sections or imports. For language models the analogue is an adversarial suffix of tokens, as in the GCG attack on aligned LLMs, which links this term to jailbreaks.

NIST AI 100-2 classifies adversarial examples as evasion attacks at deployment time, as opposed to poisoning (training time) and privacy attacks. Defences have a poor track record: many published defences relied on gradient masking and were broken by adaptive attacks (Athalye, Carlini and Wagner, 2018, "Obfuscated Gradients"). Adversarial training with PGD remains the strongest empirical defence but costs several times the normal training compute and usually reduces clean accuracy; randomised smoothing (Cohen et al., 2019) gives certified L2 guarantees only for small radii. Robustness claims should therefore be stated as a threat model plus an attack budget and evaluated with adaptive attacks, with public benchmarks such as RobustBench as reference points.