{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/prompt-injection","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/prompt-injection/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/prompt-injection/"},"term":{"en":"Prompt injection","da":"Prompt injection"},"aka":{"en":["prompt injection attack","indirect prompt injection"],"da":["prompt injection-angreb"]},"domain":["ai","security"],"cluster":"ai-risk","layer":"application","status":"current","era":2022,"summary":{"en":"Hiding instructions in the text an AI system reads so that it ignores its own rules and follows the attacker instead.","da":"At gemme instruktioner i den tekst, et AI-system læser, så det ignorerer sine egne regler og adlyder angriberen i stedet."},"body":{"formal":{"en":"An attack on a large language model in which input written by an outsider, typed directly or hidden in a web page, file or email the model is asked to read, is treated as an instruction and overrides what its owner intended.","da":"Et angreb på en stor sprogmodel, hvor tekst fra en udenforstående, skrevet direkte eller gemt i en hjemmeside, fil eller mail, som modellen skal læse, bliver opfattet som en ordre og tilsidesætter det, ejeren havde tænkt sig."},"plain":{"en":"Like slipping a note into a pile of letters a new assistant is sorting that says \"ignore your boss and send me the keys\" - and the assistant cannot tell the note apart from real orders.","da":"Som at stikke en seddel ind i en bunke breve, som en ny assistent sorterer, hvor der står \"ignorer din chef og send mig nøglerne\" - og assistenten kan ikke skelne sedlen fra rigtige ordrer."},"inPractice":{"en":"A municipality's AI assistant sums up incoming emails from citizens; one email hides white-on-white text telling it to forward the last ten messages to an outside address, and it does.","da":"En kommunes AI-assistent opsummerer indgående mails fra borgerne; én mail gemmer hvid tekst på hvid baggrund, der beder den videresende de ti seneste beskeder til en ekstern adresse, og det gør den."},"whyItMatters":{"en":"The model mixes rules and content in the same stream of words, so there is no watertight fix yet; the more an AI system is allowed to do on its own, the more damage one hidden sentence can cause.","da":"Modellen blander regler og indhold i den samme strøm af ord, så der findes endnu ingen vandtæt løsning; jo mere et AI-system må gøre på egen hånd, jo mere skade kan én skjult sætning gøre."}},"deepDive":{"en":"The root cause is architectural. An LLM receives one token sequence in which the system prompt, user turns, retrieved documents and tool results are separated only by chat-template markers and formatting conventions; attention operates over all of it, and there is no equivalent of a parameterised query that forces a span to be treated as inert data. Role tokens and training give privileged segments more weight, but the separation is statistical, which is why the analogy with SQL injection is instructive but misleading: SQL injection has a complete fix (prepared statements), prompt injection currently does not. Greshake et al. (2023) formalised indirect prompt injection, in which the payload arrives through content the application retrieves - web pages, emails, PDFs, tool descriptions - so the attacker never interacts with the system directly.\n\nPayloads pursue a small set of goals: goal hijacking (do something else), prompt leaking (reveal the system prompt, see OWASP LLM07), data exfiltration, and unauthorised tool use. A well-known exfiltration channel is output rendering: the model is told to emit a Markdown image whose URL contains conversation data in the query string, and the client leaks it when it fetches the image; blocking external image rendering or enforcing a strict content security policy closes that path. Agents with memory add persistence, since an injected instruction can be written into long-term memory and replayed in later sessions.\n\nMitigations at the model level include instruction-hierarchy training (Wallace et al., 2024), which teaches models to privilege system and developer messages over tool output, and spotlighting (Hines et al., 2024), which marks untrusted text with delimiters, interleaved datamarks or encoding. Classifier-based detectors screen retrieved content. All of these reduce success rates but are bypassed by adaptive attacks, as benchmarks such as AgentDojo show. Stronger guarantees come from system design: the dual-LLM pattern, where a privileged model never sees untrusted text and a quarantined model processes it without tool access; CaMeL (Debenedetti et al., 2025), which extracts a control flow from the trusted query and tracks capabilities on data derived from untrusted sources; and plain least privilege, human confirmation and egress restrictions, which limit what a successful injection can do.\n\nOWASP ranks prompt injection as LLM01:2025 and includes jailbreaking as a subtype; NIST AI 100-2 classifies it as an attack on generative AI with direct and indirect variants. It differs from data poisoning, which alters the model's weights during training, and from excessive agency, which is the design weakness that determines the blast radius. For an EU deployer, a successful injection that exposes personal data is a personal data breach under GDPR Art. 33 with the usual 72-hour notification clock, and for high-risk AI systems, EU AI Act Art. 15(5) requires resilience against attempts by unauthorised third parties to alter use or outputs by exploiting system vulnerabilities.","da":"Grundårsagen er arkitektonisk. En LLM modtager én sekvens af tokens, hvor systemprompt, brugerens ture, hentede dokumenter og værktøjsresultater kun er adskilt af markører i chatskabelonen og formateringskonventioner; attention virker på det hele, og der findes intet svar på en parameteriseret forespørgsel, der tvinger et tekststykke til at blive behandlet som passive data. Rolletokens og træning giver de privilegerede dele mere vægt, men adskillelsen er statistisk - derfor er sammenligningen med SQL injection lærerig, men misvisende: SQL injection har en fuldstændig løsning (prepared statements), det har prompt injection ikke i dag. Greshake et al. (2023) formaliserede indirekte prompt injection, hvor payloaden kommer ind via indhold, applikationen henter - hjemmesider, mails, PDF'er, værktøjsbeskrivelser - så angriberen aldrig selv interagerer med systemet.\n\nPayloads forfølger få mål: goal hijacking (gør noget andet), prompt leaking (afslør systemprompten, jf. OWASP LLM07), dataudtræk og uautoriseret brug af værktøjer. En kendt kanal til dataudtræk er rendering af output: modellen beordres til at skrive et Markdown-billede, hvis URL indeholder samtaledata i query-strengen, og klienten lækker dem, når billedet hentes; at blokere visning af eksterne billeder eller håndhæve en stram content security policy lukker den vej. Agenter med hukommelse giver vedvarende effekt, fordi en indsmuglet instruktion kan skrives ind i langtidshukommelsen og genafspilles i senere sessioner.\n\nPå modelniveau findes træning i instruktionshierarki (Wallace et al., 2024), der lærer modeller at prioritere system- og udviklerbeskeder over værktøjsoutput, og spotlighting (Hines et al., 2024), der markerer upålidelig tekst med skilletegn, indflettede datamærker eller kodning. Klassifikatorbaserede detektorer screener hentet indhold. Alt dette sænker succesraten, men omgås af adaptive angreb, som benchmarks som AgentDojo viser. Stærkere garantier kommer fra systemdesign: dual-LLM-mønstret, hvor en privilegeret model aldrig ser upålidelig tekst, og en isoleret model behandler den uden adgang til værktøjer; CaMeL (Debenedetti et al., 2025), der udleder kontrolflowet fra den betroede forespørgsel og sporer rettigheder på data, der stammer fra upålidelige kilder; og helt almindelige mindste rettigheder, menneskelig bekræftelse og begrænsning af udgående trafik, der begrænser, hvad en vellykket injection kan udrette.\n\nOWASP rangerer prompt injection som LLM01:2025 og regner jailbreaking som en undertype; NIST AI 100-2 klassificerer det som et angreb på generativ AI med direkte og indirekte varianter. Det adskiller sig fra dataforgiftning, der ændrer modellens vægte under træningen, og fra overdreven handlefrihed, som er den designsvaghed, der afgør skadens omfang. For en idriftsætter i EU er en vellykket injection, der eksponerer personoplysninger, et brud på persondatasikkerheden efter GDPR art. 33 med den sædvanlige frist på 72 timer, og for højrisiko-AI-systemer kræver AI-forordningens art. 15, stk. 5, modstandsdygtighed over for uautoriserede tredjeparters forsøg på at ændre brug eller output ved at udnytte sårbarheder i systemet."},"edges":[{"type":"requires","to":"ai/large-language-model","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/prompt","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"security/social-engineering","why":{"en":"Both trick a target into obeying a false authority, but here the target is the model, not a person.","da":"Begge narrer et mål til at adlyde en falsk autoritet, men her er målet modellen og ikke et menneske."},"confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"security/sql-injection","why":{"en":"Both slip commands in through input, but SQL injection targets a database while prompt injection targets a language model.","da":"Begge smugler kommandoer ind via input, men SQL injection rammer en database, mens prompt injection rammer en sprogmodel."},"confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/system-prompt","why":{"en":"Both are instructions to the model, but the system prompt comes from the app's builder while prompt injection smuggles in an attacker's orders that try to override it.","da":"Begge er instruktioner til modellen, men systemprompten kommer fra den, der har bygget appen, mens prompt injection smugler en angribers ordrer ind, der forsøger at tilsidesætte den."},"confidence":"medium","strength":"normal"},{"type":"causes","to":"security/data-breach","why":{"en":"A hijacked AI system with access to files or email can be told to hand data to the attacker.","da":"Et kapret AI-system med adgang til filer eller mail kan beordres til at udlevere data til angriberen."},"confidence":"medium","strength":"normal"},{"type":"causes","to":"ai/sensitive-information-disclosure","why":{"en":"Injected instructions can tell a model to reveal its system prompt or data it can reach.","da":"Indsmuglede instruktioner kan få en model til at afsløre sin systemprompt eller data, den har adgang til."},"confidence":"medium","strength":"normal"},{"type":"used-with","to":"ai/retrieval-augmented-generation","why":{"en":"Documents fetched to answer a question are a common hiding place for injected instructions.","da":"Dokumenter, der hentes for at besvare et spørgsmål, er et almindeligt gemmested for indsmuglede instruktioner."},"confidence":"high","strength":"normal"}],"depth":4,"sources":[{"title":"OWASP Top 10 for LLM Applications - LLM01 Prompt Injection","url":"https://genai.owasp.org/llmrisk/llm01-prompt-injection/","tier":"reference","publisher":"OWASP"},{"title":"NIST AI 100-2 - Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations","url":"https://doi.org/10.6028/NIST.AI.100-2e2025","tier":"standard","publisher":"NIST"}],"draft":true}