Reranking: a second pass that has to earn its cost
A reranker reorders a short list accurately; it cannot create what the first pass missed.

The pipeline shape is standard enough to have a name: “Retrieval: Bi-Encoder / Re-Ranker: Cross-Encoder”. The first stage compares a query against pre-computed vectors; the second stage reads the query together with each candidate and produces a relevance score. That second stage is what this sheet is about — what it can fix, what it cannot, and why it is the component most often added for the wrong reason.
Why two stages exist
The asymmetry is computational, not stylistic. A bi-encoder can embed every document in advance, because the document embedding does not depend on the query; a cross-encoder cannot, because it scores the pair — the description of that cost, in the work on late-interaction retrieval, is that the query and the document must go through the model together “to compute a single relevance score”, and that doing this has the property of increasing “computational cost by orders of magnitude”.
The consequence is the whole architecture in one sentence: retrieve widely and cheaply, then reorder narrowly and accurately. The first stage is chosen for recall over the corpus; the second is chosen for precision over a list short enough to afford it.
What the reranker adds
A score conditioned on the query. The service documentation puts it plainly: “Given a query and a list of documents, Rerank indexes the documents from most to least semantically relevant to the query.”
Two properties follow. The score is per query, so it can correct an ordering that the vector space got wrong; and it is a reordering, so it can only work on what the first stage returned. A reranker is not a retriever: it cannot recover a chunk that the first stage never found, which is why the index sheet’s distinction between a retrieval failure and an index failure matters before any reranker is added.
Where it earns its keep
- When the first stage is asked for a wide net. Retrieving the top 50 to protect recall and then handing the top 5 to the model is exactly the division of labour a reranker is for.
- When near-duplicates compete. Chunks that embed similarly but differ in one decisive detail are ordered arbitrarily by distance and meaningfully by a model that reads both.
- When the query is long or multi-part. A long question has several aspects; a query embedding averages them, while a cross-encoder can weigh the part that matters for each document.
Those three are stated as reasoning, not as measured results: they describe the mechanism, and the decision to add a reranker is a measurement on your own data.
What it costs, and what it cannot do
The source material is candid about the cost side: “Because reranking adds an extra step at runtime, it inevitably adds a small amount of latency”. To that, three costs belong at L3:
- A model to operate. A second model in the request path with its own version, its own resource needs and its own failure mode.
- A cost proportional to the candidate list. Reranking fifty chunks means fifty scored pairs, each of which carries its tokens.
- A new place for a mistake. A reranker that misjudges the top item demotes the right answer, and the generator reads the top few.
And the limit that is easy to forget: the reranker reorders. If the right passage is not in the candidate list, the second stage has nothing to promote.
It can also be worth it for a reason that has nothing to do with accuracy: the same documentation notes that reranking “provides better responses and reduces cost and latency because the model is processing less information” — a smaller, better-ordered context is cheaper to generate from, so part of the benefit shows up in the bill rather than in the answer.
How to decide, and how to check that the decision was right
Start from the failure you are actually seeing. If the right document is in the candidate list but not in the context, a reranker addresses it. If the right document is not in the candidate list at all, the problem is earlier — recall, chunking, or the query itself — and a reranker will only make the wrong list tidier.
Then measure: keep a set of queries whose correct chunk is known to be present, compare the position of that chunk with and without the second stage, and look at whether the top-n you pass to the model changed. That comparison is the only evidence that the extra model, latency and tokens bought something; the evaluation sheets of this layer describe how to build the set and how to read the result.
Level and prerequisites. L3 — integration level: the reader must be able to place a reranking stage in a retrieval pipeline, say what it can and cannot fix, and recognise when the problem is elsewhere. Prerequisites: the embeddings, vector-index and hybrid-search sheets of this layer.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Applications — the node this sheet sits in.
References
- Sentence Transformers — Retrieve & Re-Rank — the two-stage pipeline and the bi-encoder/cross-encoder division of labour.
- Khattab and Zaharia — ColBERT — the cost of scoring query–document pairs one at a time, and the late-interaction alternative.
- Cohere — Rerank overview — what a reranking API returns.
- Anthropic — Introducing Contextual Retrieval — the added latency of a reranking step, and the “processing less information” observation.