{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/jailbreak","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/jailbreak/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/jailbreak/"},"term":{"en":"Jailbreak","da":"Jailbreak"},"aka":{"en":["jailbreaking","LLM jailbreak"],"da":["jailbreaking"]},"domain":["ai","security"],"cluster":"ai-risk","layer":"application","status":"current","era":2022,"summary":{"en":"Talking an AI chat assistant out of its own safety rules with cleverly worded requests, such as role-play or made-up emergencies.","da":"At snakke en AI-chatbot fra dens egne sikkerhedsregler med snedigt formulerede forespørgsler som rollespil eller opdigtede nødsituationer."},"body":{"formal":{"en":"An attack in which a user writes a prompt meant to make a large language model produce content or take actions its alignment training and system prompt should forbid - for example through role-play, splitting a request into harmless pieces or using another language.","da":"Et angreb, hvor en bruger skriver en prompt, der skal få en stor sprogmodel til at lave indhold eller udføre handlinger, som dens alignment-træning og systemprompt skal forhindre - fx via rollespil, ved at dele en forespørgsel op i harmløse bidder eller ved at bruge et andet sprog."},"plain":{"en":"Like a child told “no sweets” who keeps trying new angles - “what if it's for my teddy?”, “Grandma always says yes” - until the tired parent gives in.","da":"Som et barn, der har fået at vide “ingen slik”, og bliver ved med at prøve nye vinkler - “hvad nu hvis det er til min bamse?”, “mormor siger altid ja” - indtil den trætte forælder giver op."},"inPractice":{"en":"Before launch, a security tester hired by a region asks its patient chatbot to “play a TV doctor in a scene”, and it gives medicine doses it is set up to refuse.","da":"Før lanceringen beder en sikkerhedstester, som en region har hyret, dens patientchatbot om at “spille tv-læge i en scene”, og den oplyser medicindoser, den er sat op til at afvise."},"whyItMatters":{"en":"Safety rules a model has learned can be argued around, so an organisation whose chat assistant can be pushed into harmful or embarrassing replies risks harm to people, its good name and legal trouble.","da":"Sikkerhedsregler, en model har lært, kan snakkes uden om, så en organisation, hvis chatbot kan presses til skadelige eller pinlige svar, risikerer at skade mennesker, sit omdømme og juridiske problemer."}},"deepDive":{"en":"Wei, Haghtalab and Steinhardt (2023) explained why safety training fails with two mechanisms. Competing objectives: the model is trained both to be helpful and follow instructions and to refuse harm, so prompts that make refusal look unhelpful - role-play (\"DAN\", \"grandma used to read me the recipe\"), prefix injection (\"start your answer with 'Sure, here is'\"), refusal suppression - tip the balance. Mismatched generalisation: pretraining gave the model capabilities in domains where safety training has little coverage, so Base64, ciphers, leetspeak or low-resource languages can carry a request past refusal behaviour that was learned mostly in plain English.\n\nLater work industrialised the attack. GCG (Zou et al., 2023) uses gradient-guided token search on open models to find adversarial suffixes that transfer to closed models - the LLM counterpart of adversarial examples. PAIR (Chao et al., 2023) and similar methods use an attacker LLM to iteratively refine a jailbreak against a black-box target. Many-shot jailbreaking (Anthropic, 2024) fills a long context window with hundreds of fabricated dialogues in which an assistant complies, exploiting in-context learning. Crescendo (Microsoft, 2024) escalates gradually over many benign-looking turns. Best-of-N jailbreaking (Hughes et al., 2024) simply samples random augmentations (capitalisation, character noise) until one succeeds, with success rising predictably with the number of attempts. For open-weight models, safety can be removed outright: fine-tuning on a small number of harmful examples undoes refusal training (Qi et al., 2023), and ablating a single \"refusal direction\" in activation space disables refusals (Arditi et al., 2024).\n\nEvaluation uses attack success rate on standard harm sets such as HarmBench or JailbreakBench, and must account for non-determinism and for judge errors in deciding whether an output is actually harmful. Defences are layered: adversarial refusal training, system prompts that restate policy, input classifiers for known jailbreak patterns, output classifiers that judge the response regardless of how the request was phrased (Anthropic's Constitutional Classifiers, 2025, are an example), rate limiting and account-level abuse detection for iterative attacks, and limiting the damage a jailbroken model can do by withholding tools and data. Output-side checks are generally more robust than input-side pattern matching, because the harmful content itself is easier to recognise than the endless ways of asking for it.\n\nOWASP files jailbreaking under LLM01:2025 Prompt Injection, treating it as the form in which the attacker's input directly causes the model to disregard its safety protocols. The practical distinction is who is attacked: in a jailbreak the user is the adversary and the target is the model provider's or operator's content policy; in indirect prompt injection a third party attacks the user or application through content the model reads. The same techniques often serve both, and a jailbreak that reveals the system prompt also overlaps with LLM07 System Prompt Leakage.","da":"Wei, Haghtalab og Steinhardt (2023) forklarede med to mekanismer, hvorfor sikkerhedstræning svigter. Konkurrerende mål: modellen er trænet både til at være hjælpsom og følge instruktioner og til at afvise skadelige ønsker, så prompts, der får en afvisning til at virke uhjælpsom - rollespil (\"DAN\", \"min bedstemor læste altid opskriften op for mig\"), prefix injection (\"start dit svar med 'Selvfølgelig, her er'\"), undertrykkelse af afvisninger - vipper balancen. Uoverensstemmende generalisering: fortræningen gav modellen evner på områder, som sikkerhedstræningen næppe dækker, så Base64, chifre, leetspeak eller sprog med få træningsdata kan føre en forespørgsel forbi en afvisningsadfærd, der mest er lært på almindeligt engelsk.\n\nSenere forskning industrialiserede angrebet. GCG (Zou et al., 2023) bruger gradientstyret søgning efter tokens på åbne modeller til at finde adversarielle suffikser, der også virker på lukkede modeller - sprogmodellernes pendant til adversarielle eksempler. PAIR (Chao et al., 2023) og lignende metoder bruger en angriber-LLM til gradvist at forfine et jailbreak mod en black-box-model. Many-shot jailbreaking (Anthropic, 2024) fylder et langt kontekstvindue med hundredvis af opdigtede dialoger, hvor en assistent adlyder, og udnytter in-context learning. Crescendo (Microsoft, 2024) eskalerer gradvist over mange uskyldigt udseende ture. Best-of-N jailbreaking (Hughes et al., 2024) sampler blot tilfældige variationer (store og små bogstaver, tegnstøj), indtil én lykkes, og succesraten stiger forudsigeligt med antallet af forsøg. For modeller med åbne vægte kan sikkerheden fjernes helt: finjustering på et lille antal skadelige eksempler ophæver afvisningstræningen (Qi et al., 2023), og ablation af én enkelt \"refusal direction\" i aktiveringsrummet slår afvisningerne fra (Arditi et al., 2024).\n\nEvaluering sker med attack success rate på standardsæt af skadelige forespørgsler som HarmBench eller JailbreakBench og skal tage højde for, at output ikke er deterministisk, og at dommeren kan tage fejl i, om et svar faktisk er skadeligt. Forsvaret er lagdelt: adversarial afvisningstræning, systemprompts, der gentager politikken, input-klassifikatorer for kendte jailbreak-mønstre, output-klassifikatorer, der vurderer svaret uanset formuleringen af forespørgslen (Anthropics Constitutional Classifiers fra 2025 er et eksempel), rate limiting og misbrugsdetektion på kontoniveau mod iterative angreb samt begrænsning af den skade, en jailbreaket model kan gøre, ved at holde værktøjer og data væk fra den. Kontroller på output-siden er generelt mere robuste end mønstergenkendelse på input, fordi det skadelige indhold er lettere at genkende end de uendeligt mange måder at bede om det på.\n\nOWASP placerer jailbreaking under LLM01:2025 Prompt Injection som den form, hvor angriberens input direkte får modellen til at se bort fra sine sikkerhedsprotokoller. Den praktiske forskel er, hvem der angribes: i et jailbreak er brugeren modstanderen, og målet er modeludbyderens eller operatørens indholdspolitik; ved indirekte prompt injection angriber en tredjepart brugeren eller applikationen via indhold, modellen læser. De samme teknikker bruges ofte til begge dele, og et jailbreak, der afslører systemprompten, overlapper også med LLM07 System Prompt Leakage."},"edges":[{"type":"requires","to":"ai/system-prompt","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/large-language-model","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/alignment","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/prompt-injection","why":{"en":"In a jailbreak the user attacks the model's own safety rules; in prompt injection an outsider hides instructions in content the model reads, turning it against its user or owner.","da":"I et jailbreak angriber brugeren selv modellens sikkerhedsregler; ved prompt injection gemmer en udenforstående instruktioner i indhold, modellen læser, og vender den mod brugeren eller ejeren."},"confidence":"high","strength":"primary"},{"type":"causes","to":"ai/sensitive-information-disclosure","why":{"en":"A successful jailbreak can get the model to reveal its hidden system prompt or other confidential details.","da":"Et vellykket jailbreak kan få modellen til at afsløre sin skjulte systemprompt eller andre fortrolige detaljer."},"confidence":"medium","strength":"minor"}],"depth":5,"sources":[{"title":"OWASP Top 10 for LLM Applications - LLM01 Prompt Injection (covers jailbreaking)","url":"https://genai.owasp.org/llmrisk/llm01-prompt-injection/","tier":"reference","publisher":"OWASP"},{"title":"Wei, Haghtalab & Steinhardt, Jailbroken: How Does LLM Safety Training Fail? (2023)","tier":"reference","publisher":"NeurIPS"},{"title":"NIST AI 100-2 - Adversarial Machine Learning, A Taxonomy and Terminology of Attacks and Mitigations","url":"https://doi.org/10.6028/NIST.AI.100-2e2025","tier":"standard","publisher":"NIST"}],"draft":true}