Systematic evaluation and regression criteria
Pick the metric, the judge and the case count before a change is allowed to ship.

An evaluation is an instrument, and every instrument has a resolution: a difference smaller than its own instability is not a result. Systematic evaluation is the discipline of choosing that instrument deliberately — metric, judge, number of cases — and writing the regression criterion down before the change it will judge. Get it wrong and every later step inherits the error.
OpenAI frames the goal directly: evaluations “test model outputs to ensure they meet style and content criteria that you specify”, and writing them is “an essential component to building reliable applications”, “especially when upgrading or trying new models” (openai-evals.txt, line 1538). The criterion is the specification; a run is only evidence about it.
Choose the metric before the judge
The first decision is what the number means. Zheng et al. name three judge designs: pairwise comparison, where the judge must “determine which one is better or declare a tie”; single-answer grading, where it scores one answer directly; and reference-guided grading, which adds a correct answer “if applicable” (llm-as-judge-2306.05685.txt, lines 226–231). Their trade-offs are stated: pairwise comparison strains “given that the number of possible pairs grows quadratically”, while single-answer grading “may become unstable, as absolute scores are likely to fluctuate more than relative pairwise results if the judge model changes” (lines 234–235, 236–237). Agreement itself has a definition you must be able to compute — “the probability of randomly selected individuals (but not identical) of each type agreeing on a randomly selected question” (lines 404–405).
The judge is an instrument with a known error
A strong judge can be good — “strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans” (lines 22–24) — and still carry documented defects: “position bias, verbosity bias, self-enhancement bias, and limited reasoning ability” (line 125). Position bias is measurable: against two near-identical answers, “Only GPT-4 outputs consistent results in more than 60% of cases” (line 291). The conservative fix exposes the instability instead of hiding it: “call a judge twice by swapping the order of two answers and only declare a win when an answer is preferred in both orders. If the results are inconsistent after swapping, we can call it a tie” (lines 339–342). With a ground truth to hand, giving the judge its own answer as a reference cut the failure rate “from 70% to 15%” (line 364). The judge is part of the instrument and must be pinned.
How many cases, and how large a difference is real
MT-bench is “a benchmark consisting of 80 high-quality multi-turn questions” (line 191), and the larger Arena study “randomly sample 3K single-turn votes from 30K arena data” (line 399). The decisive finding is resolution: agreement rises “from 70% to nearly 100%” as the models’ performance gap widens, which the authors read as the judge “aligns with humans better when significant performance differences exist between the models” (lines 432–433). Near the floor, where two versions are close, agreement approaches chance — a small gap there is the instrument’s noise, not the model’s improvement. NIST states the same limit for any deployed system: it is “demonstrated to be valid and reliable” only with the “Limitations of the generalizability beyond the conditions under which the technology was developed” documented (lines 1455–1456).
A criterion is decidable only if it is written down first
NIST MEASURE 2.3 gives the required form: “performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for conditions similar to deployment setting(s). Measures are documented.” (lines 1431–1432). For a judge-based criterion, OpenAI supplies the shape: the testing_criteria is “The graders that determine if the model output is correct” (line 2155). A decidable criterion names, in advance, the dataset, the grader, the pass threshold and the tie rule — the tie rule stops the judge from deciding the borderline case after seeing the result, the failure the swap experiment exposes. The fixed prompt is what keeps agreement comparable, which is why shared benchmarks exist: “Evaluation with publicly available prompts ensures reproducibility and comparability between papers” (lm-eval-harness-README.md, line 54). NIST also asks that what cannot be measured be documented (lines 1329–1331).
Where the methodology stops
The limits come with the design. A criterion is decidable only for the comparison it defines: change the judge, the prompt or the case set and you have changed the instrument, so an old result no longer compares to a new one. A dataset is a sample, not a specification — it can catch a regression on inputs it contains and is silent about everything else. Re-feeding recorded trajectories is the trajectory-replay sheet; running this method automatically on every change, and deciding whether it blocks a release, is the continuous-evaluation sheet. This sheet owns only the method.
Level and prerequisites. L5 — advanced systems engineering: the reader must design a measurement whose resolution is known before a change ships, choose and pin the metric and the judge, and turn a comparison into a criterion another engineer can apply without re-deriving the method. Prerequisites: the L1–L2 foundations of this area, the L3 evaluation sheets, and the L4 sheets on agent behaviour and prompting.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Operations and risk — the node this sheet sits in.
References
- Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — the three metric designs and their trade-offs, the definition of agreement, the documented biases and the swap and reference-guided mitigations, and the scale of the MT-bench and Arena studies.
- OpenAI — Working with evals — the definition of an eval, the
testing_criteriagraders, and the reason to write criteria before a change. - NIST — AI 600-1, Generative AI Profile — the MEASURE subcategories on measuring performance criteria for deployment-like conditions and on demonstrated validity and reliability.
- EleutherAI — Language Model Evaluation Harness — fixed, publicly available prompts as the basis for reproducibility and comparability.