Retrieval-augmented generation (RAG)

The original RAG model (Lewis et al., 2020) was trained end to end: a Dense Passage Retriever (DPR) bi-encoder selected passages from a Wikipedia index of about 21 million 100-word chunks, and a BART sequence-to-sequence generator conditioned on them, either on the same passages for the whole answer (RAG-Sequence) or marginalising over passages per token (RAG-Token). Today the term almost always means a looser engineering pattern in which an off-the-shelf retriever and an off-the-shelf LLM are joined only through the prompt, with no joint training.

A production pipeline has an offline and an online half. Offline: parse source formats (PDF, HTML, Office), split into chunks, commonly a few hundred tokens with some overlap and aligned to headings, attach metadata (source, date, owner, access-control list), embed each chunk and write it to a vector index, often alongside a BM25 keyword index. Online: optionally rewrite the user query, run dense and keyword retrieval in parallel, fuse the ranked lists (reciprocal rank fusion is a common choice), apply a cross-encoder reranker to the top candidates, filter by the caller's permissions, then pack the best chunks into the prompt with source identifiers and an instruction to answer only from them and cite. Agentic variants let the model issue several searches iteratively instead of one fixed retrieval.

Quality problems split cleanly into retrieval failures and generation failures, and they must be measured separately. Retrieval is evaluated with recall@k or similar on a labelled question set: if the right chunk is not retrieved, no prompt wording will fix the answer. Typical causes are bad chunk boundaries that separate a rule from its exception, tables flattened into noise, queries using different vocabulary than documents, and stale or duplicate versions outranking the current one. Generation is evaluated for faithfulness (is every claim supported by the retrieved context) and answer relevance; failures include ignoring the context in favour of parametric knowledge, merging two sources incorrectly, and answering confidently when retrieval returned nothing useful.

Security properties follow from the fact that retrieved text enters the context window with the same standing as any other text. Permission filtering must happen at query time against the user's identity, not at index time with a service account, or the assistant becomes a way to read documents the user cannot open. Any document an outsider can influence - an inbound email, a supplier PDF, a public web page - is a vector for indirect prompt injection, and poisoned documents can be crafted to rank highly for target queries. OWASP's 2025 LLM Top 10 addresses these under LLM01 Prompt Injection and LLM08 Vector and Embedding Weaknesses. Because the index is a copy of the source data, deletion and retention must propagate to it.