{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/sensitive-information-disclosure","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/sensitive-information-disclosure/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/sensitive-information-disclosure/"},"term":{"en":"Sensitive information disclosure","da":"Afsløring af følsomme oplysninger"},"aka":{"en":["LLM data leakage"],"da":["datalæk fra sprogmodeller"]},"domain":["ai","security"],"cluster":"ai-risk","layer":"data","status":"current","summary":{"en":"An AI system revealing private or secret details, such as personal data or business secrets, to someone who should not see them.","da":"Et AI-system, der afslører private eller hemmelige detaljer som personoplysninger eller forretningshemmeligheder for uvedkommende."},"body":{"formal":{"en":"A weakness, listed as LLM02 in the OWASP Top 10 for LLM Applications 2025, in which a large language model or the application around it exposes personal data, confidential business data, credentials or its system prompt through its output, whether taken from training data, retrieved documents or the conversation.","da":"En svaghed, opført som LLM02 i OWASP Top 10 for LLM Applications 2025, hvor en stor sprogmodel eller systemet omkring den udleverer personoplysninger, fortrolige forretningsdata, loginoplysninger eller sin systemprompt i sit output, hvad enten de stammer fra træningsdata, hentede dokumenter eller samtalen."},"plain":{"en":"Like a chatty receptionist who has overheard everything in the office and, asked the right question in a friendly way, repeats a colleague's home address or salary to a stranger.","da":"Som en snakkesalig receptionist, der har overhørt alt på kontoret og, når man spørger venligt på den rigtige måde, gentager en kollegas hjemmeadresse eller løn over for en fremmed."},"inPractice":{"en":"A municipality connects its new AI assistant to every shared drive; a trainee in citizen services asks what the managers earned last year and gets figures from a payroll file she has no right to open.","da":"En kommune kobler sin nye AI-assistent til alle fællesdrev; en elev i borgerservice spørger, hvad cheferne tjente sidste år, og får tal fra en lønfil, hun ikke har ret til at åbne."},"whyItMatters":{"en":"One careless answer can be a personal data breach that may have to be reported to the authorities within 72 hours under GDPR, and text that has left the system cannot be recalled.","da":"Ét uforsigtigt svar kan være et brud på persondatasikkerheden, som efter GDPR kan skulle anmeldes til myndighederne inden for 72 timer, og tekst, der har forladt systemet, kan ikke kaldes tilbage."}},"deepDive":{"en":"Disclosure has four distinct sources, and each needs a different control. Training data: models memorise, especially sequences that are duplicated in the corpus or rare and distinctive. Carlini et al. (2021) extracted verbatim personal data, including names, phone numbers and email addresses, from GPT-2 by sampling and ranking outputs by perplexity, and Nasr et al. (2023) showed a \"divergence\" attack - asking a production chat model to repeat a single word indefinitely - that made it emit memorised training text at scale. Fine-tuning data: a model fine-tuned on support tickets or case files can reproduce them, and membership inference attacks can reveal whether a specific record was in the training set. Context: anything placed in the prompt window - retrieved documents, tool results, other users' data in a shared session, the system prompt - can be echoed back. Infrastructure: logs, caches and conversation histories, as in the March 2023 incident where a caching-library bug briefly exposed other ChatGPT users' conversation titles and some billing details.\n\nIn enterprise deployments the dominant failure is not memorisation but authorisation. A RAG pipeline or copilot that indexes file shares with a service account, or that retrieves chunks without applying the querying user's access-control list, turns years of over-sharing into instant answers for anyone who asks. The fix is permission-aware retrieval - security trimming at query time against the source system's ACLs, or indexing under the user's delegated identity - plus sensitivity labels that exclude certain content from indexing altogether. OWASP separates the closely related LLM07:2025 System Prompt Leakage and stresses that the real error is putting credentials, connection strings or authorisation logic into a system prompt in the first place.\n\nControls by layer: data minimisation and PII scrubbing or pseudonymisation before training or fine-tuning; deduplication, which reduces memorisation; differentially private training (DP-SGD) where formal guarantees are needed, at a cost in utility; output filters that detect PII, secrets and CPR numbers; tenant and session isolation; retention limits and access control on prompt and response logs; and contractual terms ensuring that provider-side prompts are not used for training. Red teaming should include extraction attempts and cross-user probing.\n\nLegally, output containing personal data given to an unauthorised recipient is a personal data breach under GDPR Art. 4(12), triggering assessment and, unless unlikely to result in a risk, notification to Datatilsynet within 72 hours under Art. 33, plus notification of data subjects under Art. 34 where the risk is high. The EDPB's Opinion 28/2024 on AI models concluded that a model trained on personal data cannot automatically be treated as anonymous; anonymity must be demonstrated case by case, taking extraction and membership-inference risk into account. Disclosure is often the result of other weaknesses - prompt injection, jailbreaks or excessive agency - so it is best treated as an impact category as well as a vulnerability.","da":"Læk har fire forskellige kilder, og hver kræver sin egen kontrol. Træningsdata: modeller memorerer, især sekvenser, der optræder mange gange i korpusset, eller som er sjældne og karakteristiske. Carlini et al. (2021) trak ordrette personoplysninger, herunder navne, telefonnumre og mailadresser, ud af GPT-2 ved at sample output og rangere dem efter perplexity, og Nasr et al. (2023) viste et \"divergens\"-angreb - at bede en chatmodel i produktion gentage ét ord i det uendelige - der fik den til at udsende memoreret træningstekst i stor skala. Finjusteringsdata: en model, der er finjusteret på supportsager eller sagsakter, kan gengive dem, og membership inference-angreb kan afsløre, om en bestemt post indgik i træningssættet. Kontekst: alt, der ligger i promptvinduet - hentede dokumenter, værktøjsresultater, andre brugeres data i en delt session, systemprompten - kan gentages i svaret. Infrastruktur: logs, caches og samtalehistorik, som i hændelsen i marts 2023, hvor en fejl i et cachebibliotek kortvarigt viste andre ChatGPT-brugeres samtaletitler og visse betalingsoplysninger.\n\nI virksomheder er den dominerende fejl ikke memorering, men autorisation. En RAG-pipeline eller copilot, der indekserer fællesdrev med en servicekonto eller henter tekstbidder uden at anvende den spørgende brugers adgangskontrolliste, forvandler mange års overdeling til øjeblikkelige svar for enhver, der spørger. Løsningen er rettighedsbevidst retrieval - security trimming på forespørgselstidspunktet mod kildesystemets ACL'er eller indeksering under brugerens delegerede identitet - plus følsomhedsmærkning, der helt holder bestemt indhold ude af indekset. OWASP udskiller den nært beslægtede LLM07:2025 System Prompt Leakage og understreger, at den egentlige fejl er at lægge adgangsoplysninger, forbindelsesstrenge eller autorisationslogik i en systemprompt overhovedet.\n\nKontroller pr. lag: dataminimering og fjernelse eller pseudonymisering af personoplysninger før træning eller finjustering; deduplikering, der mindsker memorering; differentielt privat træning (DP-SGD), hvor der kræves formelle garantier, på bekostning af kvaliteten; output-filtre, der finder personoplysninger, hemmeligheder og CPR-numre; isolation mellem tenants og sessioner; opbevaringsgrænser og adgangskontrol på logs over prompts og svar; og aftalevilkår, der sikrer, at udbyderen ikke bruger prompts til træning. Red teaming bør omfatte forsøg på dataudtræk og afprøvning på tværs af brugere.\n\nJuridisk er output med personoplysninger, der gives til en uberettiget modtager, et brud på persondatasikkerheden efter GDPR art. 4, nr. 12, som udløser en vurdering og - medmindre det er usandsynligt, at bruddet indebærer en risiko - anmeldelse til Datatilsynet inden for 72 timer efter art. 33 samt underretning af de registrerede efter art. 34, hvis risikoen er høj. EDPB's udtalelse 28/2024 om AI-modeller konkluderede, at en model trænet på personoplysninger ikke automatisk kan anses for anonym; anonymitet skal påvises i hvert enkelt tilfælde under hensyn til risikoen for dataudtræk og membership inference. Læk er ofte resultatet af andre svagheder - prompt injection, jailbreaks eller overdreven handlefrihed - og behandles derfor bedst både som en konsekvenskategori og som en sårbarhed."},"edges":[{"type":"requires","to":"ai/large-language-model","confidence":"high","strength":"normal"},{"type":"requires","to":"security/personal-data","confidence":"high","strength":"normal"},{"type":"requires","to":"ai/training-data","confidence":"high","strength":"normal"},{"type":"causes","to":"security/data-breach","why":{"en":"Personal or confidential data handed to the wrong person through an AI answer is a data breach like any other leak.","da":"Personoplysninger eller fortrolige data, der via et AI-svar havner hos den forkerte, er et databrud som ethvert andet læk."},"confidence":"high","strength":"primary"}],"depth":3,"sources":[{"title":"OWASP Top 10 for LLM Applications 2025 - LLM02 Sensitive Information Disclosure","url":"https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/","tier":"reference","publisher":"OWASP"},{"title":"NIST AI 600-1 - Artificial Intelligence Risk Management Framework, Generative AI Profile (Data Privacy)","tier":"standard","publisher":"NIST"}],"draft":true}