{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/guardrails","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/guardrails/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/guardrails/"},"term":{"en":"Guardrails","da":"Guardrails (sikkerhedsværn)"},"aka":{"en":["AI guardrails","safety filters"],"da":["AI-guardrails","sikkerhedsfiltre"]},"domain":["ai"],"cluster":"ai-risk","layer":"application","status":"current","summary":{"en":"Checks placed around an AI system that block unsafe requests going in and harmful or leaking answers coming out.","da":"Kontroller rundt om et AI-system, der stopper usikre forespørgsler på vej ind og skadelige eller lækkende svar på vej ud."},"body":{"formal":{"en":"Rules, filters and small checking models that run before and after a large language model, screening input and output against a policy - blocking jailbreak attempts, removing personal data, forcing an allowed format - independently of how the model was trained.","da":"Regler, filtre og små kontrolmodeller, der kører før og efter en stor sprogmodel og tjekker input og output mod en politik - blokerer jailbreak-forsøg, fjerner personoplysninger, tvinger et tilladt format - uafhængigt af, hvordan modellen er trænet."},"plain":{"en":"Like the barriers along a mountain road - they do not steer the car, but they stop it from going over the edge when the driver makes a mistake.","da":"Som autoværnet langs en bjergvej - det styrer ikke bilen, men det forhindrer den i at køre ud over kanten, når føreren laver en fejl."},"inPractice":{"en":"A webshop's customer service manager has every chat message pass a filter that flags jailbreak wording, and every reply pass a second one that hides card numbers before the customer sees it.","da":"Kundeservicechefen i en webshop lader hver chatbesked passere et filter, der markerer jailbreak-formuleringer, og hvert svar passere et andet, der skjuler kortnumre, før kunden ser det."},"whyItMatters":{"en":"Training never makes a model fully safe, so outside checks give a second, testable line of defence that the owner can change quickly without training the model again.","da":"Træning gør aldrig en model helt sikker, så eksterne kontroller giver en ekstra forsvarslinje, der kan testes, og som ejeren hurtigt kan ændre uden at træne modellen igen."}},"deepDive":{"en":"Architecturally, guardrails are policy enforcement points in the request path of an LLM application, analogous to a WAF or DLP gateway. NVIDIA's NeMo Guardrails makes the stages explicit: input rails on the user message, retrieval rails on RAG chunks before they enter the context, dialog rails that steer conversation flow (defined in its Colang language), execution rails around tool calls, and output rails on the generated response. Each rail can reject, rewrite, redact, ask a clarifying question, or escalate to a human. Placing a check on retrieved content and tool results - not only on the user's message - is what makes guardrails relevant to indirect prompt injection.\n\nImplementations fall into three classes. Deterministic checks: regular expressions and validators for card numbers (with a Luhn check), CPR numbers, API-key formats, URL and domain allow-lists, length limits, and JSON Schema validation of structured output. Classifier models: purpose-trained safety classifiers such as Meta's Llama Guard family (an LLM fine-tuned to label prompts and responses against a hazard taxonomy, from Llama Guard 3 aligned with the MLCommons taxonomy), prompt-injection detectors such as Azure AI Content Safety Prompt Shields, and PII detectors based on named-entity recognition. LLM-as-judge: a second model prompted with a policy that grades the output. Constrained decoding - grammar- or schema-guided generation - is a related technique that prevents malformed output at generation time instead of filtering it afterwards.\n\nEvery guardrail is a classifier with a false-positive and a false-negative rate, and both matter: over-blocking pushes users to unmanaged tools, under-blocking lets attacks through. Guardrails should be evaluated on labelled test sets, including multilingual and encoded variants, with thresholds tuned per use case. Known weaknesses include obfuscation (Base64, homoglyphs, splitting a payload across turns), low-resource languages the classifier was not trained on, the classifier itself being susceptible to prompt injection when it is an LLM, and streaming, where tokens reach the user before an output check has seen the full response; mitigations are chunked checking with the ability to retract, or buffering high-risk responses. Each extra model call also adds latency and cost.\n\nGuardrails complement rather than replace alignment: alignment changes what the model tends to produce, guardrails constrain what the application accepts and emits, and they can be updated in hours without retraining. Neither replaces least privilege on tools, since a filter can be bypassed but a permission the agent does not hold cannot be abused. OWASP's LLM Top 10 recommends input and output filtering as a layer for LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure and LLM05 Improper Output Handling, and NIST AI 600-1 treats content filtering as one of several risk-management actions for generative AI.","da":"Arkitektonisk er guardrails håndhævelsespunkter for politik i requestflowet i en LLM-applikation, på linje med en WAF eller en DLP-gateway. NVIDIAs NeMo Guardrails gør trinene eksplicitte: input-rails på brugerens besked, retrieval-rails på RAG-bidder, før de kommer ind i konteksten, dialog-rails, der styrer samtalens forløb (defineret i sproget Colang), execution-rails rundt om værktøjskald og output-rails på det genererede svar. Hvert trin kan afvise, omskrive, maskere, stille et opklarende spørgsmål eller eskalere til et menneske. Det er kontrollen af hentet indhold og værktøjsresultater - ikke kun af brugerens besked - der gør guardrails relevante over for indirekte prompt injection.\n\nImplementeringerne falder i tre klasser. Deterministiske kontroller: regulære udtryk og validatorer for kortnumre (med Luhn-tjek), CPR-numre, formater for API-nøgler, allow-lists for URL'er og domæner, længdegrænser og JSON Schema-validering af struktureret output. Klassifikatormodeller: særligt trænede sikkerhedsklassifikatorer som Metas Llama Guard-familie (en LLM finjusteret til at mærke prompts og svar efter en skadestaksonomi, fra Llama Guard 3 tilpasset MLCommons' taksonomi), detektorer for prompt injection som Prompt Shields i Azure AI Content Safety og PII-detektorer baseret på named-entity recognition. LLM-as-judge: en anden model, der med en politik i prompten bedømmer svaret. Constrained decoding - generering styret af en grammatik eller et skema - er en beslægtet teknik, der forhindrer ugyldigt output allerede under genereringen i stedet for at filtrere det bagefter.\n\nEthvert guardrail er en klassifikator med en rate for falske positiver og en for falske negativer, og begge tæller: overblokering driver brugerne over i ustyrede værktøjer, underblokering lukker angreb igennem. Guardrails bør evalueres på mærkede testsæt, også med flersprogede og kodede varianter, og tærsklerne bør tilpasses den enkelte use case. Kendte svagheder er obfuskering (Base64, homoglyffer, en payload delt over flere ture), sprog med få træningsdata, som klassifikatoren ikke er trænet på, at klassifikatoren selv kan rammes af prompt injection, når den er en LLM, og streaming, hvor tokens når brugeren, før en output-kontrol har set hele svaret; modtrækket er kontrol i bidder med mulighed for at trække svaret tilbage eller buffering af højrisikosvar. Hvert ekstra modelkald giver også mere latens og flere omkostninger.\n\nGuardrails supplerer alignment frem for at erstatte det: alignment ændrer, hvad modellen er tilbøjelig til at producere, guardrails begrænser, hvad applikationen tager imod og sender ud, og de kan opdateres på timer uden gentræning. Ingen af dem erstatter mindst mulige rettigheder til værktøjer, for et filter kan omgås, men en rettighed, agenten ikke har, kan ikke misbruges. OWASP's LLM Top 10 anbefaler filtrering af input og output som et lag mod LLM01 Prompt Injection, LLM02 Sensitive Information Disclosure og LLM05 Improper Output Handling, og NIST AI 600-1 behandler indholdsfiltrering som én af flere risikostyringshandlinger for generativ AI."},"edges":[{"type":"requires","to":"ai/large-language-model","confidence":"high","strength":"normal"},{"type":"mitigates","to":"ai/prompt-injection","why":{"en":"Input checks can catch many known injection patterns and output checks can stop the harmful action, though none are watertight.","da":"Input-kontroller kan fange mange kendte injection-mønstre, og output-kontroller kan stoppe den skadelige handling, selv om ingen er vandtætte."},"confidence":"high","strength":"normal"},{"type":"mitigates","to":"ai/jailbreak","why":{"en":"Separate screens flag requests that try to talk the model out of its rules, and block harmful answers that slip through.","da":"Separate filtre markerer forespørgsler, der prøver at tale modellen fra sine regler, og blokerer skadelige svar, der slipper igennem."},"confidence":"high","strength":"primary"},{"type":"mitigates","to":"ai/sensitive-information-disclosure","why":{"en":"Output filters can find and hide personal data, secrets and other confidential details before a reply leaves the system.","da":"Output-filtre kan finde og skjule personoplysninger, hemmeligheder og andre fortrolige detaljer, før et svar forlader systemet."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"security/input-validation","why":{"en":"Guardrails carry the old habit of checking untrusted input over to free text sent to a language model.","da":"Guardrails fører den gamle vane med at tjekke upålideligt input over på fritekst, der sendes til en sprogmodel."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/human-in-the-loop","why":{"en":"When a guardrail is unsure, it can hand the request to a person to approve instead of simply blocking it.","da":"Når et guardrail er i tvivl, kan det sende forespørgslen videre til et menneske til godkendelse i stedet for blot at blokere den."},"confidence":"medium","strength":"minor"}],"depth":3,"sources":[{"title":"OWASP Top 10 for LLM Applications 2025","url":"https://genai.owasp.org/llm-top-10/","tier":"reference","publisher":"OWASP"},{"title":"NIST AI 600-1 - Artificial Intelligence Risk Management Framework, Generative AI Profile","tier":"standard","publisher":"NIST"}],"draft":true}