Sensitive information disclosure

Disclosure has four distinct sources, and each needs a different control. Training data: models memorise, especially sequences that are duplicated in the corpus or rare and distinctive. Carlini et al. (2021) extracted verbatim personal data, including names, phone numbers and email addresses, from GPT-2 by sampling and ranking outputs by perplexity, and Nasr et al. (2023) showed a "divergence" attack - asking a production chat model to repeat a single word indefinitely - that made it emit memorised training text at scale. Fine-tuning data: a model fine-tuned on support tickets or case files can reproduce them, and membership inference attacks can reveal whether a specific record was in the training set. Context: anything placed in the prompt window - retrieved documents, tool results, other users' data in a shared session, the system prompt - can be echoed back. Infrastructure: logs, caches and conversation histories, as in the March 2023 incident where a caching-library bug briefly exposed other ChatGPT users' conversation titles and some billing details.

In enterprise deployments the dominant failure is not memorisation but authorisation. A RAG pipeline or copilot that indexes file shares with a service account, or that retrieves chunks without applying the querying user's access-control list, turns years of over-sharing into instant answers for anyone who asks. The fix is permission-aware retrieval - security trimming at query time against the source system's ACLs, or indexing under the user's delegated identity - plus sensitivity labels that exclude certain content from indexing altogether. OWASP separates the closely related LLM07:2025 System Prompt Leakage and stresses that the real error is putting credentials, connection strings or authorisation logic into a system prompt in the first place.

Controls by layer: data minimisation and PII scrubbing or pseudonymisation before training or fine-tuning; deduplication, which reduces memorisation; differentially private training (DP-SGD) where formal guarantees are needed, at a cost in utility; output filters that detect PII, secrets and CPR numbers; tenant and session isolation; retention limits and access control on prompt and response logs; and contractual terms ensuring that provider-side prompts are not used for training. Red teaming should include extraction attempts and cross-user probing.

Legally, output containing personal data given to an unauthorised recipient is a personal data breach under GDPR Art. 4(12), triggering assessment and, unless unlikely to result in a risk, notification to Datatilsynet within 72 hours under Art. 33, plus notification of data subjects under Art. 34 where the risk is high. The EDPB's Opinion 28/2024 on AI models concluded that a model trained on personal data cannot automatically be treated as anonymous; anonymity must be demonstrated case by case, taking extraction and membership-inference risk into account. Disclosure is often the result of other weaknesses - prompt injection, jailbreaks or excessive agency - so it is best treated as an impact category as well as a vulnerability.