Jailbreak

Wei, Haghtalab and Steinhardt (2023) explained why safety training fails with two mechanisms. Competing objectives: the model is trained both to be helpful and follow instructions and to refuse harm, so prompts that make refusal look unhelpful - role-play ("DAN", "grandma used to read me the recipe"), prefix injection ("start your answer with 'Sure, here is'"), refusal suppression - tip the balance. Mismatched generalisation: pretraining gave the model capabilities in domains where safety training has little coverage, so Base64, ciphers, leetspeak or low-resource languages can carry a request past refusal behaviour that was learned mostly in plain English.

Later work industrialised the attack. GCG (Zou et al., 2023) uses gradient-guided token search on open models to find adversarial suffixes that transfer to closed models - the LLM counterpart of adversarial examples. PAIR (Chao et al., 2023) and similar methods use an attacker LLM to iteratively refine a jailbreak against a black-box target. Many-shot jailbreaking (Anthropic, 2024) fills a long context window with hundreds of fabricated dialogues in which an assistant complies, exploiting in-context learning. Crescendo (Microsoft, 2024) escalates gradually over many benign-looking turns. Best-of-N jailbreaking (Hughes et al., 2024) simply samples random augmentations (capitalisation, character noise) until one succeeds, with success rising predictably with the number of attempts. For open-weight models, safety can be removed outright: fine-tuning on a small number of harmful examples undoes refusal training (Qi et al., 2023), and ablating a single "refusal direction" in activation space disables refusals (Arditi et al., 2024).

Evaluation uses attack success rate on standard harm sets such as HarmBench or JailbreakBench, and must account for non-determinism and for judge errors in deciding whether an output is actually harmful. Defences are layered: adversarial refusal training, system prompts that restate policy, input classifiers for known jailbreak patterns, output classifiers that judge the response regardless of how the request was phrased (Anthropic's Constitutional Classifiers, 2025, are an example), rate limiting and account-level abuse detection for iterative attacks, and limiting the damage a jailbroken model can do by withholding tools and data. Output-side checks are generally more robust than input-side pattern matching, because the harmful content itself is easier to recognise than the endless ways of asking for it.

OWASP files jailbreaking under LLM01:2025 Prompt Injection, treating it as the form in which the attacker's input directly causes the model to disregard its safety protocols. The practical distinction is who is attacked: in a jailbreak the user is the adversary and the target is the model provider's or operator's content policy; in indirect prompt injection a third party attacks the user or application through content the model reads. The same techniques often serve both, and a jailbreak that reveals the system prompt also overlaps with LLM07 System Prompt Leakage.