Articles

Accuracy, completeness and faithfulness: three different questions

Grounded in the sources, true, and complete are three separate things to measure.

Reading: 6 minAutomation & AI

Article cover: Accuracy, completeness and faithfulness: three different questions

“Is the answer right?” hides at least three questions, and a single quality score answers none of them cleanly:

  • Faithfulness — is what the answer says supported by the material it was given?
  • Accuracy — are the statements true of the world?
  • Completeness — did the answer cover what the question required?

They are measured differently, they move in different directions when you change the system, and a pipeline that reports one number is hiding the trade-offs it is making.

Faithfulness: grounded in what was given

The definition used by the reference-free RAG metric suite is compact: “Faithfulness refers to the idea that the answer should be grounded in the given context. This is important to avoid hallucinations, and to ensure that the retrieved context can act as a justification for the generated answer.”

The way that suite computes it is worth knowing, because it is the shape most faithfulness metrics take: the answer is split into statements, each statement is checked against the context, and the score is the proportion supported — in its formal notation, the number of supported statements over the total. It is a proportion, not a verdict, and it is produced by a model being asked to verify — which is why the next sheet insists that a judge’s agreement with humans be measured rather than assumed.

The same idea appears at a different granularity in the long-form factual-precision work, which “breaks a generation into a series of atomic facts and computes the percentage of atomic facts supported by a reliable knowledge source”. Smaller units, same question: is each claim supported by something, and by what.

Two failure directions matter, and they are not opposites of each other:

  • Unfaithful — the answer adds content the context does not support. This is the failure the metric is named for.
  • Faithful and useless — every sentence is supported by the retrieved chunks, and the chunks were the wrong ones. The metric is satisfied, the user is not. That failure is a retrieval failure, and it is why faithfulness measured alone can improve while the product gets worse.

Accuracy: true of the world

Accuracy is the harder question, because “true” is not a property of the retrieved text. An answer can be perfectly faithful to an outdated document, or to a document that was always wrong, and it is then inaccurate while scoring full marks on faithfulness.

For retrieval systems, two practical proxies appear in the literature:

  • Citation precision — “each provided citation should support its associated statement”, measured as the proportion of citations that do; the audit that defined it found that a large share of citations fail this test, which is the empirical reason not to treat citations as proof.
  • Factual precision against a reliable source — the atomic-fact proportion described above, with “reliable knowledge source” as the second half of the definition, which is where the definition of reliable has to be decided explicitly rather than assumed.

Completeness: the question that reference-free metrics cannot answer alone

Completeness asks whether the answer contains what the question needed. Reference-free frameworks capture the adjacent property — the metric suite defines answer relevance as “the idea that the generated answer should address the actual question that was provided” — but relevance is not coverage: an answer can address the question and still omit half the required content.

Completeness therefore needs something to be complete against: a reference answer, or a checklist per case written when the case was created. That is an argument for the evaluation set carrying expectations (the previous sheet) rather than only inputs.

Why the three have to be reported separately

They trade against each other under the levers a team actually pulls:

Change Faithfulness Completeness Notes
Fewer, better chunks up down less to be unfaithful to, less covered
More chunks down up more noise, more material
Reranking up ~ promotes the relevant chunk, keeps the same set
Stricter “cite or refuse” instruction up down refusal is a completeness failure by design
Bigger model ~ ~ moves accuracy more than faithfulness

Those directions are reasoning about the mechanisms described in this layer, not measured results — and the point of the table is exactly that each lever has to be measured on your own set, in the three columns separately. A single score would let a team improve one and believe it had improved the product.

Level and prerequisites. L3 — integration level: the reader must be able to say which of the three questions a metric answers and to design an acceptance criterion that names the right one. The metrics themselves are measured in the evaluation sheet that follows. Prerequisites: the retrieval-failures and test-cases sheets of this layer.

Where to go next

References

  • Es et al. — RAGAS — the definitions of faithfulness, answer relevance and context relevance, the reference-free framing, and the statement-level computation.
  • Min et al. — FActScore — atomic facts and factual precision against a reliable source.
  • Liu, Zhang and Liang — Evaluating Verifiability in Generative Search Engines — citation recall and citation precision, and the requirement that a citation support its statement.
  • NIST — AI 600-1 — confabulation, the failure family that faithfulness attempts to bound.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.