Model evaluation (evals)

Model evaluation spans several layers that answer different questions. Offline evaluation scores a frozen model on held-out test sets and public benchmarks; human evaluation has experts or users rate outputs where no automatic metric is valid; LLM-as-a-judge automates part of that rating; adversarial testing and AI red teaming probe for failures an average-case metric never samples; and online evaluation (A/B tests, shadow deployments, production monitoring) measures behaviour on the real input distribution. For LLM applications an eval suite is usually code: a versioned data set of inputs, one or more graders (exact match, regex or schema checks, executing generated code, embedding similarity, model-graded rubrics) and a harness that runs in CI so that a prompt, retrieval or model change cannot ship if it regresses.

Most failures of evaluation are failures of validity rather than arithmetic. Construct validity asks whether the metric measures the property that matters; external validity asks whether the test distribution matches production, which breaks under data drift or when a demo set was hand-picked. Aggregate scores hide subgroup failures, so results should be sliced by language, customer segment, document type or protected characteristic. Scores carry sampling error and should be reported with bootstrap or binomial confidence intervals. Generative models are non-deterministic, so each item should be sampled several times; pass@k (at least one of k attempts succeeds) and pass^k (all k succeed, as used in τ-bench) answer very different reliability questions.

Governance frameworks treat evaluation as a lifecycle duty. NIST AI RMF 1.0 (NIST AI 100-1, 2023) places test, evaluation, verification and validation (TEVV) under its Measure function, and NIST AI 600-1 extends this to generative AI. ISO/IEC TS 4213:2022 specifies how to assess classification performance. The EU AI Act requires high-risk systems to be tested against "prior defined metrics and probabilistic thresholds" before being placed on the market (Art. 9(8)), to reach an appropriate level of accuracy, robustness and cybersecurity with the metrics declared in the instructions for use (Art. 15), and to be followed up by post-market monitoring (Art. 72); providers of general-purpose models with systemic risk must perform model evaluations including adversarial testing (Art. 55(1)(a)). The AI Omnibus, Regulation (EU) 2026/1744, in force since 27 July 2026, deferred the Annex III high-risk obligations to 2 December 2027 and those for high-risk AI in products under Annex I to 2 August 2028.

Evaluation must be kept separate from training: if test results drive repeated changes, the test set becomes a validation set and its score is optimistically biased. Good practice therefore freezes a test split, logs every evaluation run with model, prompt and data versions, and refreshes evaluation data when it has been seen too often. Comparing numbers across organisations is only meaningful when the harness, prompts and decoding settings are identical.