Articles

Latency, cost and resources: the bill a retrieval pipeline produces

Every retrieval step adds latency and tokens; caching is the lever and it has limits.

Reading: 6 minAutomation & AI

Article cover: Latency, cost and resources: the bill a retrieval pipeline produces

A retrieval-augmented request is more expensive than a completion, and the difference is not only the extra tokens: it is an embedding call, a search, optionally a reranking model, and a prompt that is three to twenty times longer than the question. Each step has a latency and a price, and they are not the same number — which is why the two are worth separating even when a single dashboard shows them together.

What one request spends

  • The retrieval steps — embedding the query, searching the index, optionally reranking — each with its own latency, before the model has produced anything.
  • The context tokens — every retrieved chunk is input tokens, on every request, in every conversation turn unless something is cached (below).
  • The output tokens — with the L2 measurement sheet’s warning attached: output length dominates total latency, so the token budget for the answer is also a latency budget.
  • The setup costs — the embedding pass over the corpus, the index build, and for a reranker the cost of hosting a second model. These are periodic rather than per-request, and per-request arithmetic that ignores them flatters a design that re-indexes daily.

Where the money actually goes

For self-hosted inference the governing fact is hardware utilisation: “Inference hardware utilization is very important in terms of serving costs. GPUs are expensive and we need them to do as much work as possible.” That is the economics behind the batching described in the L2 sheets: continuous batching exists so that one load of the weights serves several requests.

For hosted APIs the bill is metered per token, and cached input is the largest predictable lever. The provider’s pricing documentation states the multipliers: cache writes cost “25% more than base input tokens for 5-minute TTL”, cache reads cost “10% of base input token price”, and a longer-lived cache — “a 1-hour TTL at 2x the base input token price” — trades a higher write cost for a longer window. Those are one provider’s numbers and its own policy; the shape — cheap reads, more expensive writes, a lifetime that decides whether you win — generalises.

Caching is a design decision, not a switch

The mechanism is described as caching a prefix: “Cache prompt prefixes with cache_control to cut costs and latency, using automatic caching or explicit breakpoints with 5-minute or 1-hour TTLs.”

Three consequences follow for a retrieval pipeline:

  • Order the prompt so the stable part comes first. Instructions, tool definitions and the corpus excerpts that do not change between turns belong before the volatile question; a prefix cache can only help if the prefix is genuinely identical.
  • Short prompts do not benefit. “Shorter prompts cannot be cached, even if marked with cache_control.” A pipeline that keeps its context small has little to cache.
  • The benefit is measurable per request. The API’s usage object reports the cached portion alongside the prompt and completion tokens, so the saving can be verified rather than assumed — the same discipline the L2 measurement sheet applies to any performance claim.

For a retrieval system specifically, caching is unusually profitable: the retrieved documents are the large, stable part of the prompt within a conversation, and the user’s question is the small changing part.

Latency is not cost, and both are not throughput

  • A cached read is cheaper and faster — the two move together, which is not true in general.
  • A reranker is slower and can be cheaper: it adds a step, and it lets you pass fewer chunks to the model (the reranking sheet quotes the source making exactly this point).
  • Batching improves throughput and worsens per-user latency — the trade the L2 measurement sheet defines.
  • Retrieval adds a floor to time-to-first-token that no model tuning removes: embed, search, (rerank), then prefill.

The practical discipline is the one the L2 batch already states for generation: report latency with its conditions — model, runtime, hardware, context and concurrency — and add, for this layer, the retrieval configuration and the number of chunks in the context.

Keeping it bounded

  • Bound the request, not the month. Maximum output tokens, a maximum number of chunks, a maximum context size: each is a design decision with a quality consequence, and each is cheaper to enforce than to discover.
  • Rate limits and quotas protect the system from its own clients — the L2 API sheet classifies them; the risk of not having them is OWASP’s LLM10, Unbounded Consumption, where excessive inference “expands on what was previously Denial of Service to include risks around resource management”.
  • Attribute the cost. Per-request usage records let a team say which feature, which tenant or which query pattern consumes the budget — the difference between engineering and arguing about the invoice.

Level and prerequisites. L3 — integration level: the reader must be able to state what a request costs, in tokens, latency and money, and to choose the levers that bound it. No price list is quoted as current and no system is instrumented here. Prerequisites: the L2 measurement sheet, and the retrieval sheets of this layer that decide how many chunks there are to pay for.

Where to go next

References

  • Databricks — LLM Inference Performance Engineering — hardware utilisation as the cost driver, and the latency metrics used by the L2 sheet.
  • Anthropic — Prompt caching — the prefix-caching mechanism, the TTL options, the write/read multipliers, the minimum prompt requirement and the usage reporting.
  • OpenAI — API description — the usage object’s cached-token field.
  • OWASP — Top 10 for LLM Applications (2025), LLM10 — unbounded consumption as a resource risk.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.