Articles

Regressions across models, prompts and versions

A pipeline can get worse without anyone changing it; only a versioned baseline notices.

Reading: 6 minAutomation & AI

Article cover: Regressions across models, prompts and versions

A pipeline that answered well last month can answer worse today with nobody having edited it. The model was updated, the index grew, a dependency changed, the prompt template was reformatted by a library upgrade. “It worked yesterday” is not evidence about today, and the only mechanism that notices is a baseline you can re-run.

What can change under you

Four things, and only the first is obvious:

  • The model — a hosted model updated by its provider, a version string that now resolves elsewhere, a local weights file replaced (the L2 batch’s placement sheet covers the difference between a file you keep and a model someone else maintains).
  • The prompt — an edit, a template applied differently, a system message shortened.
  • The retrieval side — a new document that now outranks an old one, a re-index with a different chunk size, a filter changed.
  • The plumbing — an SDK upgrade that changes defaults, a timeout that truncates a longer context, a token limit that cuts a document’s tail.

Each of those can move quality in either direction, and none of them announces itself. A regression test is a claim about a version, repeated over versions.

More than accuracy

The argument for measuring several things comes from the benchmark that made it a design principle: it measures “7 metrics (accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency) for each of 16 core scenarios”, covering 87.5% of the possible pairs, explicitly to keep “metrics beyond accuracy” from falling by the wayside and to expose “trade-offs across models and metrics”.

An application’s version of that argument is narrower but structurally the same: a system can keep its answers and lose its latency, or improve its faithfulness and double its cost. The metrics to keep pinned are therefore not only quality metrics — latency, cost per request and refusal rate belong to the baseline too, because they are the numbers a “harmless” prompt edit most often moves.

Reproducibility is part of the measurement

The public harness states the discipline in one sentence: “Evaluation with publicly available prompts ensures reproducibility and comparability between papers.” Its published prompts are pinned so that two runs mean the same thing; its configuration is a versioned file.

For a pipeline, the same discipline has four pinned artefacts, and a result that is missing one of them is not comparable with anything — including a later run of itself:

  1. the model (and its version or file hash),
  2. the prompt (as text, not as “the current template”),
  3. the index (which documents, which chunking, when built),
  4. the evaluation set (and its revision).

The judge is part of the system

If the regression test uses a model to judge answers, the judge can change too — it is hosted, versioned and updated like everything else. Two consequences:

  • Treat the judge as a pinned dependency with its own version recorded next to the four artefacts above.
  • Know its measured agreement with human judgement, and re-check it when it changes: the LLM-as-a-judge work reports agreement with human preferences at the level humans achieve with each other for a strong judge, and names the failure modes — “position, verbosity, and self-enhancement biases, as well as limited reasoning ability” — which are exactly the distortions that show up as a regression when the judge is swapped.

A small, human-checked subset that never changes is the cheapest instrument for detecting a judge that has drifted: run it against the old and the new judge and compare the verdicts.

When a regression appears

  1. Confirm it is a regression and not noise. Re-run. An evaluation set whose result moves on its own is not yet a measurement (the test-cases sheet).
  2. Localise it. Retrieval or generation? Component metrics first, end-to-end second — the diagnosis order of the failures sheet.
  3. Check what changed, against the four pinned artefacts. The change is usually someone else’s update, which is why the versions are recorded.
  4. Then decide — revert, adjust, or accept with a written reason and a date. Accepting a regression deliberately is a legitimate outcome; accepting it silently is how quality decays.

Level and prerequisites. L3 — integration level: the reader must be able to set up a versioned baseline and to interpret a change in it. No harness is built here and no threshold is set (the next sheet covers acceptance). Prerequisites: the test-cases and evaluation sheets of this layer.

Where to go next

References

  • Liang et al. — HELM — multi-metric measurement as a design principle, with its metric list and coverage.
  • EleutherAI — LM Evaluation Harness — pinned public prompts for reproducibility, and versioned configuration.
  • Zheng et al. — Judging LLM-as-a-Judge — judge agreement with humans and the biases a judge exhibits.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.