Embedding model

Most retrieval embedding models are bi-encoders: the same transformer (or two towers) encodes query and document independently, and the per-token hidden states are pooled into one fixed-length vector, by mean pooling, the [CLS] token, or for decoder-based models the final token. Independence is the point, because document vectors can be computed once at indexing time and only the query needs encoding at search time. Output dimensions typically range from 384 to over 4,000. Models trained with Matryoshka representation learning (Kusupati et al., 2022) can be truncated to a prefix of their dimensions, trading accuracy for storage, provided the truncated vectors are renormalised.

Training is contrastive. Given a query and a relevant passage, an InfoNCE loss pushes their similarity above that of negatives: L = −log(exp(s(q, p⁺)/τ) / Σ exp(s(q, pᵢ)/τ)), where s is usually cosine similarity and τ a temperature. Other passages in the same batch serve as cheap "in-batch negatives", so large batches help, and mined hard negatives (passages that look relevant but are not) matter most for quality. Sentence-BERT (Reimers and Gurevych, 2019) showed that fine-tuning BERT in a siamese setup turned it into a usable similarity model; later families such as E5, BGE and GTE add a weakly supervised pretraining stage on hundreds of millions of mined text pairs before fine-tuning on labelled data. Many models are asymmetric and expect prefixes or instructions, such as "query: " and "passage: " for E5; omitting them at query time is a common, silent cause of poor recall. Since 2023, embedding models built on decoder-only LLMs have reached the top of public leaderboards, so the older rule that embedders are encoders is no longer reliable.

MTEB (Muennighoff et al., 2023) evaluates models across task types such as retrieval, classification, clustering, reranking and semantic textual similarity in many languages, but leaderboard averages are English-heavy and can be inflated by training on benchmark-adjacent data. For Danish, the Scandinavian Embedding Benchmark (Enevoldsen et al., NeurIPS 2024) covers Danish, Swedish and Norwegian tasks and found notable gaps between models that MTEB averages did not reveal. The reliable test remains retrieval recall on a labelled sample of one's own queries and documents.

Operationally, vectors from different models, or different versions of one model, live in incompatible spaces, so the model identifier and version should be stored with every vector, and a migration means re-embedding the corpus, often by dual-writing to a new index before cutting over. Embeddings are not anonymisation: Vec2Text (Morris et al., 2023) recovered 92 % of 32-token inputs exactly from their embeddings, so vectors derived from personal data should be treated as personal data. Neighbouring designs are learned sparse models such as SPLADE, which output vocabulary-weighted vectors for inverted indexes, and multi-vector models such as ColBERT, which keep one vector per token.