Benchmark vs LLM-as-a-judge

Why they differ

A benchmark checks answers against fixed correct ones; an LLM judge grades open answers where no single correct one exists.

Benchmark

Evaluation & metrics

A shared, public set of tasks with a fixed way of scoring, so that different models can be compared on the same terms.

Formal

A published collection of tasks with known correct answers and a fixed scoring method, used across labs to rank models; well-known ones test broad school knowledge or the fixing of real faults in code.

In plain English

Like a standard fitness course that every athlete runs, with the same hurdles and the same stopwatch, so their times can be put side by side.

In practice

An IT architect in a ministry compares five large language models by their published benchmark scores, picks two, and then tests both on 300 real questions from the ministry's own staff before choosing.

Why it matters

Benchmarks drive how models are sold and ranked, but a model can be tuned to the questions, or have seen them in its training data, so a high score does not prove it will do well on your work.

LLM-as-a-judge

Evaluation & metrics

Using one large language model to grade the answers of another against a written scoring guide, instead of paying people to read them all.

Formal

A form of model evaluation in which a large language model receives a question, one or more answers and grading instructions, and returns a score or a preference; it is checked against human grades for agreement.

In plain English

Like a talent show where one seasoned judge scores every act from a printed sheet - quick and mostly fair, but the judge has favourites of their own.

In practice

A product owner at an unemployment insurance fund changes the member chat's system prompt, has a judge model compare 1,000 old and new answers side by side, and reads only the cases where the judge is unsure.

Why it matters

It makes checking open answers cheap enough to do on every change, but judges favour longer answers, the first answer shown and their own style, and can be fooled by instructions hidden in an answer.

Shared connections

Model evaluation (evals)