{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/ai-red-teaming","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/ai-red-teaming/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/ai-red-teaming/"},"term":{"en":"AI red teaming","da":"AI red teaming"},"aka":{"en":["red teaming of AI systems"],"da":["red teaming af AI-systemer"]},"domain":["ai","security"],"cluster":"ai-risk","layer":"governance","status":"emerging","era":2023,"summary":{"en":"Testers attack an AI system on purpose, before and after release, to find ways it can be tricked into harmful or unsafe behaviour.","da":"Testere angriber med vilje et AI-system før og efter lancering for at finde måder, det kan narres til skadelig eller usikker adfærd."},"body":{"formal":{"en":"A structured, authorised exercise in which people, often helped by automated tools, act as attackers against an AI model or product to find harmful output, prompt injection, rule breaking, data leaks and other failures, and report them so they can be fixed.","da":"En struktureret, godkendt øvelse, hvor personer, ofte hjulpet af automatiske værktøjer, optræder som angribere mod en AI-model eller et AI-produkt for at finde skadeligt output, prompt injection, regelbrud, datalæk og andre fejl og rapportere dem, så de kan rettes."},"plain":{"en":"Like hiring clever troublemakers to spend a week trying to talk a new shop assistant into breaking every rule, so you learn where the training falls short before real customers do.","da":"Som at hyre snedige ballademagere til at bruge en uge på at få en ny ekspedient til at bryde alle regler, så du opdager hullerne i oplæringen, før rigtige kunder gør."},"inPractice":{"en":"Before a municipality opens a chat assistant for citizens, a team spends two weeks trying to make it reveal other citizens' case details, give wrong advice about benefits or ignore its rules, and each trick found is blocked.","da":"Før en kommune åbner en chatassistent for borgerne, bruger et team to uger på at få den til at afsløre andre borgeres sagsoplysninger, give forkerte råd om ydelser eller ignorere sine regler, og hvert trick, de finder, bliver lukket."},"whyItMatters":{"en":"AI systems fail in ways ordinary functional tests never try, and the EU AI Act requires this kind of attack testing for the largest general-purpose models, whose failures could cause harm across society.","da":"AI-systemer fejler på måder, almindelige funktionstest aldrig afprøver, og EU's AI-forordning kræver den slags angrebstest af de største AI-modeller til almen brug, hvis fejl kan skade hele samfundet."}},"deepDive":{"en":"AI red teaming borrows the military and security term but covers a broader failure space than a conventional penetration test. The targets are the model (harmful content, jailbreaks, bias, hallucination, memorised training data, dangerous capabilities such as chemical, biological or cyber uplift), the application around it (system prompt leakage, direct and indirect prompt injection, insecure handling of model output, excessive agency in tools and agents), and the infrastructure (model endpoints, retrieval stores, supply chain). The OWASP GenAI Red Teaming Guide (2025) structures the work along these layers, and MITRE ATLAS provides an ATT&CK-style matrix of adversarial ML tactics and techniques that can be used to scope and report findings.\n\nA typical engagement starts with a threat model and a harm taxonomy: which actors, which assets, which content categories and actions are unacceptable for this deployment. Testers then combine manual probing - role-play, persona and hypothetical framing, multi-turn escalation, encoding tricks, low-resource languages, poisoned documents placed where a RAG pipeline will retrieve them - with automated generation and scoring. Open-source harnesses such as Microsoft PyRIT, NVIDIA garak and promptfoo send large attack corpora, mutate prompts with an attacker LLM and grade responses with classifiers or an LLM judge. Because outputs are stochastic, results should be reported as attack success rates over many samples at fixed decoding settings, not as single screenshots.\n\nFor EU providers of general-purpose AI models with systemic risk, EU AI Act Art. 55(1)(a) requires model evaluation \"including conducting and documenting adversarial testing\" with a view to identifying and mitigating systemic risks, and the General-Purpose AI Code of Practice (July 2025) describes how signatories evidence this in its Safety and Security chapter. For high-risk systems, Art. 15 requires resilience against attempts to alter use or performance by exploiting vulnerabilities, including adversarial examples and data poisoning, which red teaming helps demonstrate. NIST AI 600-1 lists red teaming among the suggested actions for measuring generative AI risks.\n\nCommon pitfalls: testing only the base model and not the deployed configuration with its system prompt, tools and retrieval; treating a red-team pass as proof of safety rather than a lower bound on what an attacker finds; running a one-off exercise before launch while model versions, prompts and connectors change weekly; and failing to feed findings into regression test suites and guardrail rules. Microsoft's 2025 report on red teaming over 100 generative AI products stresses that many real failures come from simple techniques and system integration, not from gradient-based attacks. Red teaming complements, and does not replace, benchmark evaluation, a conventional penetration test of the surrounding infrastructure, and monitoring in production.","da":"AI red teaming låner begrebet fra militæret og sikkerhedsverdenen, men dækker et bredere fejlrum end en klassisk penetrationstest. Målene er modellen (skadeligt indhold, jailbreaks, bias, hallucinationer, memoriserede træningsdata, farlige kapabiliteter som hjælp til kemiske, biologiske eller cyberangreb), applikationen omkring den (læk af systemprompten, direkte og indirekte prompt injection, usikker håndtering af modellens output, overdreven handlefrihed i værktøjer og agenter) og infrastrukturen (model-endpoints, retrieval-lagre, forsyningskæde). OWASP GenAI Red Teaming Guide (2025) strukturerer arbejdet efter disse lag, og MITRE ATLAS giver en ATT&CK-lignende matrix over taktikker og teknikker mod maskinlæring, som kan bruges til at afgrænse og rapportere fund.\n\nEt typisk forløb starter med en trusselsmodel og en skadestaksonomi: hvilke aktører, hvilke aktiver og hvilke indholdskategorier og handlinger der er uacceptable for netop denne løsning. Testerne kombinerer derefter manuel afprøvning - rollespil, personaer og hypotetiske rammer, eskalering over flere ture, kodningstricks, sprog med få træningsdata, forgiftede dokumenter placeret, hvor en RAG-pipeline vil hente dem - med automatisk generering og scoring. Open source-værktøjer som Microsoft PyRIT, NVIDIA garak og promptfoo sender store samlinger af angreb, muterer prompts med en angriber-LLM og bedømmer svarene med klassifikatorer eller en LLM-dommer. Fordi output er stokastisk, bør resultater rapporteres som attack success rate over mange kørsler med faste decoding-indstillinger, ikke som enkelte skærmbilleder.\n\nFor udbydere i EU af AI-modeller til almen brug med systemisk risiko kræver AI-forordningens art. 55, stk. 1, litra a, modelevaluering, \"herunder gennemførelse og dokumentation af adversarial testing\", med henblik på at identificere og afbøde systemiske risici, og adfærdskodeksen for AI-modeller til almen brug (juli 2025) beskriver i kapitlet om Safety and Security, hvordan tiltrædende udbydere dokumenterer det. For højrisikosystemer kræver art. 15 modstandsdygtighed over for forsøg på at ændre brug eller ydeevne ved at udnytte sårbarheder, herunder adversarielle eksempler og dataforgiftning, hvilket red teaming er med til at påvise. NIST AI 600-1 nævner red teaming blandt de foreslåede handlinger til at måle risici i generativ AI.\n\nTypiske faldgruber: kun at teste basismodellen og ikke den konfiguration, der faktisk kører med systemprompt, værktøjer og retrieval; at behandle en bestået red team-øvelse som bevis på sikkerhed i stedet for en nedre grænse for, hvad en angriber finder; at køre én enkelt øvelse før lancering, mens modelversioner, prompts og connectors ændres hver uge; og ikke at føre fundene ind i regressionstest og guardrail-regler. Microsofts rapport fra 2025 om red teaming af over 100 generative AI-produkter understreger, at mange reelle fejl skyldes simple teknikker og systemintegration, ikke gradientbaserede angreb. Red teaming supplerer - men erstatter ikke - benchmark-evaluering, en klassisk pentest af den omgivende infrastruktur og overvågning i drift."},"edges":[{"type":"requires","to":"ai/artificial-intelligence","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/ai-governance","confidence":"high","strength":"normal"},{"type":"part-of","to":"ai/model-evaluation","why":{"en":"Deliberate attacks on a model are one of the ways its safety is checked, alongside scores, benchmarks and human review.","da":"Bevidste angreb på en model er en af måderne, dens sikkerhed kontrolleres på, sammen med tal, benchmarks og menneskelig gennemgang."},"confidence":"medium","strength":"normal"},{"type":"contrasts-with","to":"security/penetration-test","why":{"en":"A pentest looks for ways into systems and data; AI red teaming also hunts for harmful, false or rule-breaking answers from the model itself.","da":"En pentest leder efter veje ind til systemer og data; AI red teaming jagter også skadelige, falske eller regelbrydende svar fra selve modellen."},"confidence":"high","strength":"primary"},{"type":"mitigates","to":"ai/prompt-injection","why":{"en":"Testers try many hidden and direct instructions so weak spots are found and blocked before attackers find them.","da":"Testerne prøver mange skjulte og direkte instruktioner, så svage punkter bliver fundet og lukket, før angribere finder dem."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/jailbreak","why":{"en":"Red teamers try jailbreaks on purpose to find where a model's safety rules give way before attackers do.","da":"Red teamere forsøger bevidst jailbreaks for at finde, hvor en models sikkerhedsregler giver efter, før angribere gør det."},"confidence":"medium","strength":"normal"}],"depth":1,"sources":[{"title":"NIST AI 600-1 - Artificial Intelligence Risk Management Framework, Generative AI Profile","url":"https://doi.org/10.6028/NIST.AI.600-1","tier":"standard","publisher":"NIST"},{"title":"OWASP GenAI Red Teaming Guide","url":"https://genai.owasp.org/resource/genai-red-teaming-guide/","tier":"reference","publisher":"OWASP"},{"title":"Regulation (EU) 2024/1689 (Artificial Intelligence Act), Article 55","url":"https://eur-lex.europa.eu/eli/reg/2024/1689/oj","tier":"standard","publisher":"European Union"}],"draft":true}