Continuous evaluation in the pipeline
Run evaluations on every change, and decide which ones may block a release.

A pipeline that evaluates continuously answers two questions on every change: did this change make the system worse, and is that difference large enough to stop the release? The first is measurement; the second is a gate. Running both on every commit is what “continuous” adds, and it moves the hard problem to deciding which checks may block — because an untrusted gate is overridden until it means nothing.
A gate is a measurement plus a decision rule, and the measurement is noisy. A model’s output varies between runs, an LLM judge has its own biases, and production traffic carries no reference answer. A sound pipeline runs cheap, exact checks on everything, samples the expensive ones, and refuses to read a single run as a verdict.
Offline before the ship, online after it
Evaluation has two placements. Offline evaluation runs a curated dataset during development “to compare versions, benchmark performance, and catch regressions”; online evaluation reads live traffic “to detect issues and measure quality on live traffic” (LangSmith Evaluation, lines 69 and 73). The offline run is the gate: it holds reference answers and can score every case on every change. The online run is detection: production has no labels, so it asserts only what needs no reference — safety, format validation, quality heuristics, a reference-free judge — on traces (line 141). The loop closes when a failing trace is promoted into the dataset: “create targeted evaluators, validate fixes with offline experiments, and redeploy” (line 153). The dataset is the memory; the run is the reflex.
Deterministic checks and judged checks
An evaluator is a code rule or a model. A string check compares output to a labelled reference — “use a string_check grader to evaluate the output” (OpenAI Working with evals, line 2490) — fast, exact and repeatable, so it belongs on the critical path. An LLM-as-judge scores open-ended quality where no exact answer exists; strong judges “can match both controlled and crowdsourced human preferences well,” and do so “achieving over 80% agreement, the same level of agreement between humans” (Zheng et al., lines 23 and 24). The same study documents what the judge brings — “position bias, verbosity bias, self-enhancement bias, and limited reasoning ability” (line 125) — defects that do not surface in the score they produce. The two belong at different points.
What runs on every change, and what is sampled
Cost sets the cadence, and cadence sets what the gate can catch. A deterministic grader over pinned cases can run on every commit; a judged evaluation over a large dataset cannot, because it scales with the sample, and a check that slows every push gets disabled. Production states the pattern plainly: online evaluators take “filters and sampling rates to control costs” (LangSmith Evaluation, line 141) — only a fraction of traffic is judged. The real decision is cadence, not evaluator: a cheap check that always runs catches more than an expensive one that almost never does. Standard benchmarks buy comparison only within the capability they test — a harness running “a large number of different evaluation tasks” gives “reproducibility and comparability between papers” (LM Evaluation Harness README, lines 44 and 54).
When a gate is trustworthy
A gate is trustworthy only in proportion to the change it resolves. One run against one threshold is fragile: the run-to-run variance of a sampled model can exceed the effect a change introduces. The sound shape is a baseline, a candidate and a tolerance, run with enough repetitions that a real regression clears the noise floor while sampling jitter does not trip it. A judged metric is noisier than a deterministic one, so its tolerance is wider and its failure softer; a deterministic check makes a trustworthy gate because its result does not move on unchanged input. The common failure is exactness about what the product does not care about — a string match on free-text, or a judge score tracked to two decimals — yielding failures nobody acts on.
What a red gate should mean
Blocking a release is a policy that should follow the confidence behind the check. A deterministic failure — a broken schema, an absent required string, a tripped safety rule — blocks, with no ambiguity to weigh. A judged regression above tolerance blocks rarely and more often opens a review, because the judge is a signal, not a verdict. A sampled online alert does not block a release already shipped; it opens an incident and adds a case to the dataset. Flakiness is the gate’s worst enemy: a check that fails on unchanged code gets overridden, and a gate that is routinely overridden has stopped being one. Continuous evaluation is worth its cost only when its red is rare, believed and acted on.
Level and prerequisites. L5 — advanced systems engineering: the reader places eval stages along the delivery path, budgets which checks run per commit and which are sampled, and sets the blocking rule for a measurement noisier than a compiler error. Prerequisites: the L1–L2 foundations of this area and the L3 evaluation sheets on what to measure. Metric, judge and comparison-criteria methodology belongs to the systematic-evaluation sheet, not here.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Operations & risk — the node this sheet sits in.
References
- OpenAI — Working with evals — the evals concept and the string-check grader.
- LangChain / LangSmith — LangSmith Evaluation — offline versus online placement, sampled online evaluators, and the trace-to-dataset feedback loop.
- EleutherAI — Language Model Evaluation Harness (README) — the unified multi-task framework and the reproducibility of public benchmarks.
- Zheng et al. — Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685) — judge agreement with human preference, and the biases that come with it.