{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/benchmark","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/benchmark/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/benchmark/"},"term":{"en":"Benchmark","da":"Benchmark"},"aka":{"en":["benchmark suite","benchmark dataset"],"da":["benchmark-suite","benchmarkdatasæt"]},"domain":["ai"],"cluster":"evaluation","layer":"model","status":"current","summary":{"en":"A shared, public set of tasks with a fixed way of scoring, so that different models can be compared on the same terms.","da":"Et fælles, offentligt sæt opgaver med en fast måde at give point på, så forskellige modeller kan sammenlignes på lige vilkår."},"body":{"formal":{"en":"A published collection of tasks with known correct answers and a fixed scoring method, used across labs to rank models; well-known ones test broad school knowledge or the fixing of real faults in code.","da":"En offentliggjort samling opgaver med kendte rigtige svar og en fast pointmetode, som bruges på tværs af laboratorier til at rangere modeller; kendte eksempler tester bred skoleviden eller evnen til at rette rigtige fejl i kode."},"plain":{"en":"Like a standard fitness course that every athlete runs, with the same hurdles and the same stopwatch, so their times can be put side by side.","da":"Som en fast forhindringsbane, som alle atleter løber, med de samme forhindringer og det samme ur, så deres tider kan stilles op ved siden af hinanden."},"inPractice":{"en":"An IT architect in a ministry compares five large language models by their published benchmark scores, picks two, and then tests both on 300 real questions from the ministry's own staff before choosing.","da":"En IT-arkitekt i et ministerium sammenligner fem store sprogmodeller ud fra deres offentliggjorte benchmarkresultater, udvælger to og tester dem derefter på 300 rigtige spørgsmål fra ministeriets egne medarbejdere, før der vælges."},"whyItMatters":{"en":"Benchmarks drive how models are sold and ranked, but a model can be tuned to the questions, or have seen them in its training data, so a high score does not prove it will do well on your work.","da":"Benchmarks styrer, hvordan modeller sælges og rangeres, men en model kan tunes til spørgsmålene eller have set dem i sine træningsdata, så en høj score beviser ikke, at den klarer sig godt i jeres arbejde."}},"deepDive":{"en":"A benchmark fixes three things: a data set of task instances, a protocol for presenting them to a model (prompt template, number of few-shot examples, decoding settings, tools allowed) and a scoring function. Classic examples show the range: MMLU (Hendrycks et al., 2021) is multiple-choice across 57 subjects scored by accuracy; HumanEval (Chen et al., 2021) has 164 hand-written Python problems scored by pass@k against unit tests; SWE-bench (Jimenez et al., 2023) contains 2,294 real GitHub issues from 12 Python repositories, graded by whether the model's patch makes the repository's tests pass, and SWE-bench Verified is a 500-instance subset screened by human engineers because some originals were underspecified or had unfair tests. Chatbot Arena instead ranks models from crowdsourced pairwise votes fitted with a Bradley-Terry model.\n\nThe protocol matters as much as the data. Changing the prompt format, the number of shots, whether the answer is scored by generated letter or by the log-likelihood of each option, or the evaluation harness version can move a score by several points, so numbers from different model cards are often not comparable. HELM (Liang et al., 2022) was designed partly to address this by running many models under one standardised protocol and reporting several metrics per scenario (accuracy, calibration, robustness, fairness, efficiency) instead of a single number.\n\nBenchmarks decay. Saturation happens when top models approach the ceiling, as GLUE did soon after its 2018 release, prompting SuperGLUE in 2019. Contamination happens because public test items get scraped into pretraining corpora; mitigations include n-gram overlap checks, unique canary strings embedded in benchmark files so their presence in a corpus can be detected and filtered, private held-out splits and continually refreshed item pools. Adaptive overfitting arises when a community repeatedly tunes against one public test set: Recht et al. (2019) rebuilt CIFAR-10 and ImageNet test sets following the original procedures and saw accuracy drops of 3% to 15% and 11% to 14% respectively, although model rankings largely held. Label noise adds another layer: Northcutt et al. (2021) estimated at least 3.3% label errors on average across ten widely used test sets.\n\nIn practice a benchmark score is a prior, not a verdict. It narrows the candidate list, after which a task-specific evaluation on in-domain data, with the organisation's own prompts, context and failure costs, decides. A benchmark differs from a private test set mainly in being public and shared, which gives comparability at the price of leakage, and from LLM-as-a-judge in having reference answers or executable checks rather than a model's opinion.","da":"Et benchmark fastlægger tre ting: et datasæt af opgaver, en protokol for, hvordan de præsenteres for modellen (promptskabelon, antal few-shot-eksempler, dekodningsindstillinger, tilladte værktøjer), og en scoringsfunktion. Klassiske eksempler viser spændvidden: MMLU (Hendrycks m.fl., 2021) er multiple choice inden for 57 fag og scores med nøjagtighed; HumanEval (Chen m.fl., 2021) har 164 håndskrevne Python-opgaver, der scores med pass@k mod unittests; SWE-bench (Jimenez m.fl., 2023) rummer 2.294 rigtige GitHub-issues fra 12 Python-repositories og bedømmes på, om modellens patch får repositoriets tests til at bestå, og SWE-bench Verified er en delmængde på 500 opgaver, som menneskelige udviklere har gennemgået, fordi nogle af de oprindelige var underspecificerede eller havde urimelige tests. Chatbot Arena rangerer i stedet modeller ud fra crowdsourcede parvise stemmer, der tilpasses med en Bradley-Terry-model.\n\nProtokollen betyder lige så meget som data. Ændrer man promptformatet, antallet af eksempler, om svaret scores på det genererede bogstav eller på log-sandsynligheden for hver mulighed, eller versionen af evalueringsværktøjet, kan scoren flytte sig flere point, så tal fra forskellige modelkort ofte ikke kan sammenlignes. HELM (Liang m.fl., 2022) blev bl.a. lavet for at løse det ved at køre mange modeller under én standardiseret protokol og rapportere flere mål pr. scenarie (nøjagtighed, kalibrering, robusthed, fairness, effektivitet) i stedet for ét tal.\n\nBenchmarks forældes. Mætning opstår, når de bedste modeller nærmer sig loftet, som GLUE gjorde kort efter lanceringen i 2018, hvilket førte til SuperGLUE i 2019. Kontaminering opstår, fordi offentlige testopgaver skrabes med i fortræningsdata; modtræk er n-gram-overlapstjek, unikke canary strings i benchmarkfilerne, så deres tilstedeværelse i et korpus kan opdages og filtreres fra, private tilbageholdte splits og løbende fornyede opgavepuljer. Adaptiv overtilpasning opstår, når et helt forskningsfelt tuner mod det samme offentlige testsæt: Recht m.fl. (2019) genopbyggede testsættene til CIFAR-10 og ImageNet efter de oprindelige procedurer og så fald i nøjagtighed på 3-15 % og 11-14 %, selv om modellernes indbyrdes rangorden stort set holdt. Fejl i labels kommer oveni: Northcutt m.fl. (2021) anslog i gennemsnit mindst 3,3 % forkerte labels på tværs af ti udbredte testsæt.\n\nI praksis er en benchmarkscore et udgangspunkt, ikke en dom. Den indsnævrer feltet, hvorefter en opgavespecifik evaluering på egne data, med organisationens egne prompts, kontekst og fejlomkostninger, afgør valget. Et benchmark adskiller sig fra et privat testsæt ved at være offentligt og fælles, hvilket giver sammenlignelighed på bekostning af lækage, og fra LLM som dommer ved at have referencesvar eller eksekverbare tjek i stedet for en models vurdering."},"edges":[{"type":"requires","to":"ai/test-set","why":{"en":"A benchmark is a test set made public and shared, which makes scores comparable but lets its questions leak into training data.","da":"Et benchmark er et testsæt, der er gjort offentligt og fælles, hvilket gør resultaterne sammenlignelige, men lader spørgsmålene sive ind i træningsdata."},"confidence":"high","strength":"primary"},{"type":"part-of","to":"ai/model-evaluation","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/llm-as-a-judge","why":{"en":"A benchmark checks answers against fixed correct ones; an LLM judge grades open answers where no single correct one exists.","da":"Et benchmark tjekker svar mod faste rigtige svar; en LLM-dommer bedømmer åbne svar, hvor der ikke findes ét rigtigt."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/model-card","why":{"en":"Model cards usually list a model's benchmark scores so buyers can compare it with others.","da":"Modelkort lister typisk en models benchmarkresultater, så købere kan sammenligne den med andre."},"confidence":"high","strength":"normal"}],"depth":3,"sources":[{"title":"Hendrycks et al. (2021), Measuring Massive Multitask Language Understanding","url":"https://arxiv.org/abs/2009.03300","tier":"reference","publisher":"ICLR 2021"},{"title":"Liang et al. (2022), Holistic Evaluation of Language Models","url":"https://arxiv.org/abs/2211.09110","tier":"reference","publisher":"Stanford CRFM"},{"title":"NIST AI 600-1, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile","url":"https://doi.org/10.6028/NIST.AI.600-1","tier":"standard","publisher":"NIST"},{"title":"Recht et al. (2019), Do ImageNet Classifiers Generalize to ImageNet?","url":"https://proceedings.mlr.press/v97/recht19a.html","tier":"reference","publisher":"ICML 2019 (PMLR 97)"},{"title":"Northcutt, Athalye & Mueller (2021), Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks","url":"https://arxiv.org/abs/2103.14749","tier":"reference","publisher":"NeurIPS 2021 Datasets and Benchmarks"}],"draft":true}