Embedding

Two different things share the name. Inside every language model there is a token-embedding table, a learned matrix of shape vocabulary size by model dimension, from which each input token ID selects one row; these vectors are only the starting point and are transformed layer by layer into contextual representations. Separately, embedding models (text encoders) map a whole passage to one fixed-length vector, typically a few hundred to a few thousand dimensions, by pooling the final hidden states (mean pooling or a dedicated CLS or end-of-sequence token). Retrieval, clustering, deduplication and classification use the second kind.

The lineage runs from word2vec (Mikolov et al., 2013), whose skip-gram and CBOW objectives produced static word vectors with the famous linear analogies, through contextual encoders such as BERT (2018), to sentence encoders trained contrastively: pairs of related texts (question and answer, title and body, paraphrases) are pulled together and in-batch negatives pushed apart with an InfoNCE-style loss. Sentence-BERT (2019) made this bi-encoder setup practical; benchmarks such as MTEB compare models across retrieval, clustering and similarity tasks. Some newer models are trained so that a prefix of the vector (for example the first 256 of 1,024 dimensions) remains usable, which lets operators trade accuracy for storage.

Similarity is usually measured by cosine similarity or, for L2-normalised vectors, the equivalent dot product. Scores are only comparable within one model: vectors from different models, or different versions of the same model, live in unrelated spaces, so changing the embedding model means re-embedding the whole corpus. At scale, exact search is replaced by approximate nearest-neighbour indexes such as HNSW or IVF with product quantisation, which trade a little recall for large speed gains. A bi-encoder compresses a passage before seeing the query, so it misses fine distinctions such as negation or exact identifiers; that is why pipelines add keyword search (hybrid search) and a cross-encoder reranker that reads query and passage together.

Embeddings are not anonymisation. Morris et al. (2023) showed that an iterative inversion method (vec2text) recovered 92% of 32-token inputs exactly from their embeddings, and personal names could be recovered from embeddings of clinical notes. Under GDPR an embedding of personal data therefore remains personal data, a vector store needs the same access control, retention and deletion as the source documents, and a leaked index should be treated like a leak of the text. OWASP's 2025 LLM Top 10 covers related risks as LLM08 Vector and Embedding Weaknesses.