Evaluating RAG and tool calling: measuring the pipeline, not the model
Component metrics, model judges and human checks each measure a different part.

A retrieval pipeline has more parts than a model, and each part has its own failure. Evaluating “the system” with one score therefore answers a question nobody can act on: when the number drops, the team still does not know whether to change the chunking, the index, the prompt or the model.
This sheet is about the three layers of measurement that make a pipeline’s failures attributable, and about what each of them costs.
Layer one: components
Component metrics measure one step, in isolation, against a known expectation:
- Retrieval — the failure rate already used in this layer: the proportion of questions whose answering passage is not in the top k, computed from the known location of the answer. It needs nothing from a model.
- Context relevance — the retrieved material should be on topic; the RAG metric suite defines it as “the idea that the retrieved context should be focused, containing as little irrelevant information as possible” and notes the cost argument for it: long context passages are expensive to feed to a model.
- Tool selection and arguments — the analogue for tools: was the right tool chosen, were the arguments schema-valid, and did the system decline to call when it should? The criteria are easy to state and easy to label by hand on a small set, which is what makes tool behaviour measurable at all.
Component metrics are also the cheapest, because most of them need no generation: a retrieval failure rate is computed from the retrieval result alone.
Layer two: end-to-end answers
Here the metrics are the ones the previous sheet distinguished — faithfulness, accuracy, completeness, citation precision — computed on the output. The relevant design decision is what they are computed against:
- Reference-free. The RAG suite states the reason its metrics exist: “we usually do not have access to human-annotated datasets or reference answers. We therefore focus on metrics that are fully self-contained and reference-free.” The metric is defined against the retrieved context and the question, not against a golden answer.
- Reference-based. Where a reference exists (a curated case with a known answer), comparison against it is more direct — and it is also what makes completeness measurable at all.
The price of reference-free metrics is that they are model judgements, and the next section applies.
Layer three: who judges
Model-based judges. The automation is stated plainly in the RAG evaluation literature: “Model-based evaluation is an inexpensive strategy to test generative output quality”. ARES is the worked example of the design — an “Automated RAG Evaluation System, for evaluating RAG systems along the dimensions of context relevance, answer faithfulness, and answer relevance”, which “finetunes lightweight LM judges” on synthetic training data and then corrects their predictions with “a small set of human-annotated datapoints” using prediction-powered inference, needing “only a few hundred human annotations during evaluation”. Two claims from that work are worth carrying: judges “remain effective across domain shifts”, and the alternative — building labelled sets by hand — “demand[s] high expertise and impose[s] considerable annotation costs”.
Human judges. The reference, and the scarce resource: human evaluation is “time-consuming and costly”, and human preferences are “otherwise very expensive to obtain”. What humans give that no metric does is the definition of what “good” means for this product; what they cannot give is repetition at scale.
The bridge between them is measurement, not trust. A model judge is a component with a version and a failure mode: the RAG work reports how its metrics agreed with human annotators (“in around 90% of the cases” on its dataset), and the LLM-as-a-judge work verifies judgements against expert votes and crowdsourced preferences, reporting agreement at the level of human-to-human agreement for a strong judge while naming the biases to watch. An unvalidated judge is an opinion with an API key.
Offline and online
The two phases answer different questions, and a serious pipeline has both: the offline mode is described as “Run evaluations on curated datasets during development to compare versions, benchmark performance, and catch regressions” — test before you ship — while online evaluation runs against live traffic, with the connection between them made deliberately: “Add real traces to a dataset so a failure you saw once becomes a test you run every time.”
Offline evaluation is repeatable and narrow; online evaluation is realistic and noisy. The cases that matter travel from one to the other.
The order to build it in
- Decide what wrong means for this product (the previous sheet’s three questions).
- Build a small set with expectations and failure labels.
- Measure retrieval separately from generation — the cheapest metrics, and the ones that explain most end-to-end failures.
- Pick metrics that can be computed from what you have: reference-free where no reference exists, reference-based where a curated case provides one.
- Validate any model judge against a human-checked subset, and record its agreement.
- Run it offline on every change, and use online results to grow the set.
That order front-loads the cheap and local measures, which is what makes a later end-to-end failure diagnosable rather than merely visible.
Level and prerequisites. L3 — integration level: the reader must be able to plan an evaluation for a pipeline, not just for a model. The tools and the harness are a Progetti concern; the acceptance thresholds are the next sheet. Prerequisites: the metrics of the previous sheets of this layer.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Operations and risk — the node this sheet sits in.
References
- Es et al. — RAGAS — reference-free metrics, the context-relevance definition, and the reported agreement with human annotators.
- Saad-Falcon et al. — ARES — the automated RAG evaluation design, the synthetic-data judges, the small human-annotation budget, and the domain-shift result.
- Zheng et al. — Judging LLM-as-a-Judge — agreement with human preferences, and the biases a judge exhibits.
- Min et al. — FActScore — why human evaluation is the reference and why it is scarce.
- LangSmith — Evaluation concepts — offline and online evaluation, and turning traces into cases.