Regressions across models, prompts and versions
A pipeline can get worse without anyone changing it; only a versioned baseline notices.

A pipeline that answered well last month can answer worse today with nobody having edited it. The model was updated, the index grew, a dependency changed, the prompt template was reformatted by a library upgrade. “It worked yesterday” is not evidence about today, and the only mechanism that notices is a baseline you can re-run.
What can change under you
Four things, and only the first is obvious:
- The model — a hosted model updated by its provider, a version string that now resolves elsewhere, a local weights file replaced (the L2 batch’s placement sheet covers the difference between a file you keep and a model someone else maintains).
- The prompt — an edit, a template applied differently, a system message shortened.
- The retrieval side — a new document that now outranks an old one, a re-index with a different chunk size, a filter changed.
- The plumbing — an SDK upgrade that changes defaults, a timeout that truncates a longer context, a token limit that cuts a document’s tail.
Each of those can move quality in either direction, and none of them announces itself. A regression test is a claim about a version, repeated over versions.
More than accuracy
The argument for measuring several things comes from the benchmark that made it a design principle: it measures “7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios”, covering 87.5% of the possible pairs, explicitly to keep “metrics beyond accuracy” from falling by the wayside and to expose “trade-offs across models and metrics”.
An application’s version of that argument is narrower but structurally the same: a system can keep its answers and lose its latency, or improve its faithfulness and double its cost. The metrics to keep pinned are therefore not only quality metrics — latency, cost per request and refusal rate belong to the baseline too, because they are the numbers a “harmless” prompt edit most often moves.
Reproducibility is part of the measurement
The public harness states the discipline in one sentence: “Evaluation with publicly available prompts ensures reproducibility and comparability between papers.” Its published prompts are pinned so that two runs mean the same thing; its configuration is a versioned file.
For a pipeline, the same discipline has four pinned artefacts, and a result that is missing one of them is not comparable with anything — including a later run of itself:
- the model (and its version or file hash),
- the prompt (as text, not as “the current template”),
- the index (which documents, which chunking, when built),
- the evaluation set (and its revision).
The judge is part of the system
If the regression test uses a model to judge answers, the judge can change too — it is hosted, versioned and updated like everything else. Two consequences:
- Treat the judge as a pinned dependency with its own version recorded next to the four artefacts above.
- Know its measured agreement with human judgement, and re-check it when it changes: the LLM-as-a-judge work reports agreement with human preferences at the level humans achieve with each other for a strong judge, and names the failure modes — “position, verbosity, and self-enhancement biases, as well as limited reasoning ability” — which are exactly the distortions that show up as a regression when the judge is swapped.
A small, human-checked subset that never changes is the cheapest instrument for detecting a judge that has drifted: run it against the old and the new judge and compare the verdicts.
When a regression appears
- Confirm it is a regression and not noise. Re-run. An evaluation set whose result moves on its own is not yet a measurement (the test-cases sheet).
- Localise it. Retrieval or generation? Component metrics first, end-to-end second — the diagnosis order of the failures sheet.
- Check what changed, against the four pinned artefacts. The change is usually someone else’s update, which is why the versions are recorded.
- Then decide — revert, adjust, or accept with a written reason and a date. Accepting a regression deliberately is a legitimate outcome; accepting it silently is how quality decays.
Level and prerequisites. L3 — integration level: the reader must be able to set up a versioned baseline and to interpret a change in it. No harness is built here and no threshold is set (the next sheet covers acceptance). Prerequisites: the test-cases and evaluation sheets of this layer.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Operations and risk — the node this sheet sits in.
References
- Liang et al. — HELM — multi-metric measurement as a design principle, with its metric list and coverage.
- EleutherAI — LM Evaluation Harness — pinned public prompts for reproducibility, and versioned configuration.
- Zheng et al. — Judging LLM-as-a-Judge — judge agreement with humans and the biases a judge exhibits.