{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/keyword-search","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/keyword-search/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/keyword-search/"},"term":{"en":"Keyword search","da":"Nøgleordssøgning"},"aka":{"en":["lexical search","full-text search","BM25"],"da":["leksikalsk søgning","fritekstsøgning","BM25"]},"domain":["ai"],"cluster":"retrieval","layer":"application","status":"current","summary":{"en":"Finding texts that contain the same words as the question, ranked by how often those words appear and how rare they are overall.","da":"At finde tekster, der indeholder de samme ord som spørgsmålet, rangeret efter hvor ofte ordene står der, og hvor sjældne de er i alt."},"body":{"formal":{"en":"A way of searching that scores each stored text by the words it shares with the question, giving more weight to words that are rare across all texts while each extra repeat of a word in one text adds less than the one before; BM25 is the most common scoring recipe.","da":"En søgemetode, der giver hver gemt tekst point for de ord, den deler med spørgsmålet, med mere vægt til ord, der er sjældne på tværs af alle tekster, mens hver ekstra gentagelse af et ord i samme tekst tæller mindre end den forrige; BM25 er den mest udbredte pointformel."},"plain":{"en":"Like the index at the back of a book - look up \"tax\" and it lists every page where that exact word is printed, but not the page that says \"duty\".","da":"Som stikordsregistret bag i en bog - slå op under \"skat\", og det viser hver side, hvor netop det ord står, men ikke siden, der siger \"afgift\"."},"inPractice":{"en":"An IT support worker at a region pastes error code 4031 from the patient record system into the IT knowledge base, and keyword search returns the one article that mentions that exact code.","da":"En IT-supporter i en region indsætter fejlkode 4031 fra journalsystemet i IT-afdelingens vidensbase, og nøgleordssøgning finder den ene artikel, der nævner netop den kode."},"whyItMatters":{"en":"It is fast, cheap and easy to explain, and it wins on names, product numbers and codes, where meaning-based search tends to treat near matches as good enough.","da":"Den er hurtig, billig og let at forklare, og den vinder på navne, varenumre og koder, hvor betydningsbaseret søgning har det med at tage noget, der ligner, for godt nok."}},"deepDive":{"en":"The core data structure is the inverted index: a term dictionary (in Lucene, compressed as a finite-state transducer) mapping each term to a postings list of document IDs, usually with term frequencies and positions, stored delta-encoded and compressed. A query looks up the postings for each query term and scores only documents that appear in at least one list. Positions enable phrase and proximity queries; per-field indexes enable restricting or weighting fields such as title versus body. Top-k evaluation avoids scoring every match through dynamic pruning algorithms such as WAND and Block-Max WAND, which skip documents whose maximum possible score cannot enter the current top k.\n\nBM25, from the Okapi system at City University London and formalised in the probabilistic relevance framework (Robertson and Zaragoza, 2009), scores a document D for query Q as Σ IDF(q) · f(q, D) · (k₁ + 1) / (f(q, D) + k₁ · (1 − b + b · |D| / avgdl)). The term-frequency component saturates: the first occurrences of a term add most, and further repetitions add less and less, with k₁ controlling how fast (typical values 1.2-2.0). The parameter b (typically 0.75) normalises for document length relative to the average avgdl, so long documents are not rewarded simply for containing more words. Lucene's IDF is ln(1 + (N − n + 0.5) / (n + 0.5)) for N documents of which n contain the term, so rare terms dominate. Lucene made BM25 its default similarity in version 6.0 (2016), replacing classic TF-IDF, with k₁ = 1.2 and b = 0.75; BM25F extends the formula to weighted fields. PostgreSQL's built-in ts_rank, by contrast, is not BM25 and uses no corpus-wide IDF.\n\nQuality depends as much on the analysis chain as on the formula: tokenisation, lowercasing, Unicode folding, stop words, stemming or lemmatisation, and synonyms. Danish needs particular care. The Snowball Danish stemmer handles inflection, but productive compounding means that \"sygedagpengeloven\" does not match \"sygedagpenge\" or \"loven\" without dictionary-based decompounding, and the characters æ, ø and å must not be folded to ASCII in a way that merges distinct words. The same analyser must be applied at index and query time, otherwise terms silently fail to match.\n\nThe classic weakness is vocabulary mismatch: relevant documents that use different words score zero. Mitigations include synonym lists, query expansion with pseudo-relevance feedback (for example RM3), fuzzy matching by edit distance for typos, and, today, combining the lexical ranking with dense retrieval in hybrid search. Its strengths are explainability (every score decomposes into per-term contributions), no training requirement, cheap incremental updates and deletions, and robustness on unseen domains, which is why BM25 remains the standard baseline in retrieval benchmarks.","da":"Den centrale datastruktur er det inverterede indeks: en termordbog (i Lucene komprimeret som en finite-state transducer), der knytter hver term til en postingliste med dokument-ID'er, som regel med termfrekvenser og positioner, gemt deltakodet og komprimeret. En forespørgsel slår postingerne op for hver term og scorer kun dokumenter, der optræder på mindst én liste. Positioner muliggør frase- og nærhedsforespørgsler; indeks pr. felt gør det muligt at begrænse til eller vægte felter som titel frem for brødtekst. Top-k-evaluering undgår at score hvert match ved hjælp af dynamiske beskæringsalgoritmer som WAND og Block-Max WAND, der springer dokumenter over, hvis højest mulige score ikke kan nå ind i den aktuelle top k.\n\nBM25, der stammer fra Okapi-systemet ved City University London og er formaliseret i den probabilistiske relevansramme (Robertson og Zaragoza, 2009), scorer et dokument D for forespørgslen Q som Σ IDF(q) · f(q, D) · (k₁ + 1) / (f(q, D) + k₁ · (1 − b + b · |D| / avgdl)). Termfrekvensdelen mættes: De første forekomster af en term bidrager mest, og yderligere gentagelser bidrager mindre og mindre, hvor k₁ styrer hvor hurtigt (typiske værdier 1,2-2,0). Parameteren b (typisk 0,75) normaliserer for dokumentlængden i forhold til gennemsnittet avgdl, så lange dokumenter ikke belønnes blot for at indeholde flere ord. Lucenes IDF er ln(1 + (N − n + 0,5) / (n + 0,5)) for N dokumenter, hvoraf n indeholder termen, så sjældne termer dominerer. Lucene gjorde BM25 til standard-similarity i version 6.0 (2016) i stedet for klassisk TF-IDF, med k₁ = 1,2 og b = 0,75; BM25F udvider formlen til vægtede felter. PostgreSQL's indbyggede ts_rank er derimod ikke BM25 og bruger ingen IDF på tværs af korpusset.\n\nKvaliteten afhænger lige så meget af analysekæden som af formlen: tokenisering, små bogstaver, Unicode-foldning, stopord, stemming eller lemmatisering og synonymer. Dansk kræver særlig omhu. Snowballs danske stemmer håndterer bøjninger, men den produktive orddannelse betyder, at \"sygedagpengeloven\" ikke matcher \"sygedagpenge\" eller \"loven\" uden ordbogsbaseret opsplitning af sammensatte ord, og æ, ø og å må ikke foldes til ASCII på en måde, der slår forskellige ord sammen. Den samme analyzer skal bruges ved indeksering og forespørgsel, ellers matcher termer ikke, uden at nogen opdager det.\n\nDen klassiske svaghed er ordforrådsmismatch: Relevante dokumenter, der bruger andre ord, scorer nul. Afhjælpning omfatter synonymlister, udvidelse af forespørgslen med pseudo-relevance feedback (fx RM3), fuzzy match efter redigeringsafstand ved stavefejl og i dag kombination af den leksikalske rangering med dense retrieval i hybrid søgning. Styrkerne er forklarlighed (hver score kan opdeles i bidrag pr. term), intet behov for træning, billige løbende opdateringer og sletninger samt robusthed på ukendte domæner, og derfor er BM25 stadig standardbasislinjen i benchmarks for informationssøgning."},"edges":[{"type":"contrasts-with","to":"ai/semantic-search","why":{"en":"Keyword search needs the same words to appear; semantic search matches on meaning, so each catches results the other misses.","da":"Nøgleordssøgning kræver, at de samme ord optræder; semantisk søgning matcher på betydning, så hver fanger resultater, den anden overser."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/retrieval-augmented-generation","why":{"en":"Many RAG systems still find their passages, fully or partly, by matching words, especially for exact terms and codes.","da":"Mange RAG-systemer finder stadig helt eller delvist deres afsnit ved at matche ord, især for præcise begreber og koder."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/reranking","confidence":"high","strength":"normal"}],"depth":0,"sources":[{"title":"Robertson & Zaragoza (2009), The Probabilistic Relevance Framework - BM25 and Beyond","tier":"reference","publisher":"Foundations and Trends in Information Retrieval"},{"title":"Manning, Raghavan & Schütze, Introduction to Information Retrieval","tier":"textbook","publisher":"Cambridge University Press"}],"draft":true}