Articles

Retrieval failures and noisy context: when the wrong text is worse than none

A pipeline can fail before the model runs: wrong chunk, missing chunk, or too much noise.

Reading: 6 minAutomation & AI

Article cover: Retrieval failures and noisy context: when the wrong text is worse than none

A retrieval pipeline can fail in four places, and only one of them is the model’s fault. The chunk that answers the question may never have been retrieved; the right chunk may have been retrieved and ignored; the context may be full of plausible near-misses; or the model may have misread what it was given. Separating these is the difference between fixing a system and guessing at it.

Why this is measurable at all

The useful move is to define a failure rate rather than to argue about answer quality: measure how often the passage that answers the question is not among the top-k retrieved chunks. That is what the contextual retrieval work does when it reports a “retrieval failure rate” computed as the complement of recall at twenty documents — an objectively checkable property, because the answer’s location is known for the test set.

With that metric, the same source reports what a preprocessing change buys: contextual embeddings “reduced the top-20-chunk retrieval failure rate by 35% (5.7% → 3.7%)”, and combining contextual embeddings with contextual BM25 “reduced the top-20-chunk retrieval failure rate by 49% (5.7% → 2.9%)” — improvements obtained without touching the model or the prompt. The metric itself is stated as “1 minus recall@20”, which is what makes the figure checkable at all.

Two lessons generalise beyond that experiment:

  • Retrieval quality is measurable separately from generation quality, which means a pipeline’s problems can be localised instead of attributed to “the AI”.
  • Preprocessing and retrieval choices move the number, which is why the earlier sheets of this layer — chunking, embeddings, hybrid search, reranking — are the first place to look when answers are wrong.

The four failures, and how they look from outside

1. Never retrieved (recall). The answering passage is not in the candidate list. Symptoms: the model answers from general knowledge; the answer is confidently wrong; adding the right document to the corpus changes nothing because the query never surfaces it. Causes live in chunking, embedding choice, filters and the index’s recall setting (the index sheet).

2. Retrieved and ignored (position). The passage is in the context but in the region models attend to least: long-context studies find the worst performance when the evidence sits “in the middle of long contexts”. Symptoms: the answer ignores a document that is visibly present; reordering the same chunks changes the answer, which is the cheapest experiment available.

3. Drowned in noise (precision). The context contains many topically similar but non-answering chunks. The model has to choose, and nothing in the pipeline has made the choice easy. Symptoms: answers that blend several documents; hedged or vague responses; a difference between what a human reading the same five chunks would conclude and what the model said. Reranking and smaller contexts address this.

4. Misread (faithfulness). The right passage is present, well-placed and clean, and the model still gets it wrong — for instance by consolidating two numbers, or by attributing a statement to the wrong source. NIST’s generative-AI profile has a name for the family this belongs to: confabulation, “the production of confidently stated but erroneous or false content”. This is the failure class that evaluation sheets measure, and it does not disappear because retrieval was good.

Why the wrong text is worse than no text

A pipeline with no retrieved context fails visibly: the model answers from training, and the answer is generic. A pipeline that retrieves a plausible but wrong passage can produce an answer that reads like documented fact — with a citation attached. The audit of generative search engines quantifies the result for citation-bearing answers: on average, “a mere 51.5%” of generated sentences were fully supported by their citations, and “only 74.5% of citations support their associated statement”.

The operational consequence is a design principle worth stating explicitly: a pipeline should be able to answer “I could not find it”, and the path that produces that answer should be as ordinary as the path that produces a confident one. A system that always answers is a system whose failure mode is invention.

Retrieved text is also untrusted input

Everything in the corpus is text the pipeline did not write, and it arrives in the same request as the instructions. Two consequences belong to this level:

  • Delimit and label it. A retrieved chunk must be visibly a quoted source, never an instruction the model could mistake for its own configuration (the assembly sheet).
  • Assume it can carry instructions. Documents can contain text aimed at the model rather than at a human reader — the injection class. Analysing and containing that is L4 material (prompt injection, tool permissions, approval); at this level the design rule is that retrieved content never acquires authority merely by being retrieved.

OWASP’s vector-and-embedding entry covers the same surface from the data side: inadequate access controls lead to unauthorized access to embeddings, and the pipeline’s own ingestion path is a place where data can be altered before it is ever retrieved.

What to do when an answer is wrong: the order of diagnosis

  1. Was the answering passage retrieved? If not → recall: chunking, embedding, filters, index settings.
  2. Was it in the context, and where? If it moved the answer when reordered → assembly and position.
  3. What else was in the context? If the block was full of near-misses → reranking, smaller context.
  4. Was the passage read correctly? If yes to all of the above → faithfulness, and an evaluation set: the model is the problem, and a prompt or model change is the fix to test.
  5. Was the source itself wrong or outdated? If so, the pipeline behaved correctly and the corpus did not.

That order is also the cheapest: each step is a query or a reordering, and only the last one costs a model change.

Level and prerequisites. L3 — integration level: the reader must be able to localise a retrieval failure and to name the mechanism in play. The measurement itself is developed in the evaluation sheets of this layer; the security containment of untrusted retrieved text is L4. Prerequisites: the earlier sheets of this layer.

Where to go next

References

  • Anthropic — Introducing Contextual Retrieval — the retrieval failure rate as a metric and the measured effect of contextual embeddings with contextual BM25.
  • Liu et al. — Lost in the Middle — where relevant material is used inside a long context.
  • Liu, Zhang and Liang — Evaluating Verifiability in Generative Search Engines — the citation audit figures.
  • NIST — AI 600-1 — the definition of confabulation.
  • OWASP — Top 10 for LLM Applications (2025), LLM08 — vector and embedding weaknesses.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.