Articles

Reranking: a second pass that has to earn its cost

A reranker reorders a short list accurately; it cannot create what the first pass missed.

Reading: 5 minAutomation & AI

Article cover: Reranking: a second pass that has to earn its cost

The pipeline shape is standard enough to have a name: “Retrieval: Bi-Encoder / Re-Ranker: Cross-Encoder”. The first stage compares a query against pre-computed vectors; the second stage reads the query together with each candidate and produces a relevance score. That second stage is what this sheet is about — what it can fix, what it cannot, and why it is the component most often added for the wrong reason.

Why two stages exist

The asymmetry is computational, not stylistic. A bi-encoder can embed every document in advance, because the document embedding does not depend on the query; a cross-encoder cannot, because it scores the pair — the description of that cost, in the work on late-interaction retrieval, is that the query and the document must go through the model together “to compute a single relevance score”, and that doing this has the property of increasing “computational cost by orders of magnitude”.

The consequence is the whole architecture in one sentence: retrieve widely and cheaply, then reorder narrowly and accurately. The first stage is chosen for recall over the corpus; the second is chosen for precision over a list short enough to afford it.

What the reranker adds

A score conditioned on the query. The service documentation puts it plainly: “Given a query and a list of documents, Rerank indexes the documents from most to least semantically relevant to the query.”

Two properties follow. The score is per query, so it can correct an ordering that the vector space got wrong; and it is a reordering, so it can only work on what the first stage returned. A reranker is not a retriever: it cannot recover a chunk that the first stage never found, which is why the index sheet’s distinction between a retrieval failure and an index failure matters before any reranker is added.

Where it earns its keep

  • When the first stage is asked for a wide net. Retrieving the top 50 to protect recall and then handing the top 5 to the model is exactly the division of labour a reranker is for.
  • When near-duplicates compete. Chunks that embed similarly but differ in one decisive detail are ordered arbitrarily by distance and meaningfully by a model that reads both.
  • When the query is long or multi-part. A long question has several aspects; a query embedding averages them, while a cross-encoder can weigh the part that matters for each document.

Those three are stated as reasoning, not as measured results: they describe the mechanism, and the decision to add a reranker is a measurement on your own data.

What it costs, and what it cannot do

The source material is candid about the cost side: “Because reranking adds an extra step at runtime, it inevitably adds a small amount of latency”. To that, three costs belong at L3:

  • A model to operate. A second model in the request path with its own version, its own resource needs and its own failure mode.
  • A cost proportional to the candidate list. Reranking fifty chunks means fifty scored pairs, each of which carries its tokens.
  • A new place for a mistake. A reranker that misjudges the top item demotes the right answer, and the generator reads the top few.

And the limit that is easy to forget: the reranker reorders. If the right passage is not in the candidate list, the second stage has nothing to promote.

It can also be worth it for a reason that has nothing to do with accuracy: the same documentation notes that reranking “provides better responses and reduces cost and latency because the model is processing less information” — a smaller, better-ordered context is cheaper to generate from, so part of the benefit shows up in the bill rather than in the answer.

How to decide, and how to check that the decision was right

Start from the failure you are actually seeing. If the right document is in the candidate list but not in the context, a reranker addresses it. If the right document is not in the candidate list at all, the problem is earlier — recall, chunking, or the query itself — and a reranker will only make the wrong list tidier.

Then measure: keep a set of queries whose correct chunk is known to be present, compare the position of that chunk with and without the second stage, and look at whether the top-n you pass to the model changed. That comparison is the only evidence that the extra model, latency and tokens bought something; the evaluation sheets of this layer describe how to build the set and how to read the result.

Level and prerequisites. L3 — integration level: the reader must be able to place a reranking stage in a retrieval pipeline, say what it can and cannot fix, and recognise when the problem is elsewhere. Prerequisites: the embeddings, vector-index and hybrid-search sheets of this layer.

Where to go next

References

  • Sentence Transformers — Retrieve & Re-Rank — the two-stage pipeline and the bi-encoder/cross-encoder division of labour.
  • Khattab and Zaharia — ColBERT — the cost of scoring query–document pairs one at a time, and the late-interaction alternative.
  • Cohere — Rerank overview — what a reranking API returns.
  • Anthropic — Introducing Contextual Retrieval — the added latency of a reranking step, and the “processing less information” observation.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.