Latency, cost and resources: the bill a retrieval pipeline produces
Every retrieval step adds latency and tokens; caching is the lever and it has limits.

A retrieval-augmented request is more expensive than a completion, and the difference is not only the extra tokens: it is an embedding call, a search, optionally a reranking model, and a prompt that is three to twenty times longer than the question. Each step has a latency and a price, and they are not the same number — which is why the two are worth separating even when a single dashboard shows them together.
What one request spends
- The retrieval steps — embedding the query, searching the index, optionally reranking — each with its own latency, before the model has produced anything.
- The context tokens — every retrieved chunk is input tokens, on every request, in every conversation turn unless something is cached (below).
- The output tokens — with the L2 measurement sheet’s warning attached: output length dominates total latency, so the token budget for the answer is also a latency budget.
- The setup costs — the embedding pass over the corpus, the index build, and for a reranker the cost of hosting a second model. These are periodic rather than per-request, and per-request arithmetic that ignores them flatters a design that re-indexes daily.
Where the money actually goes
For self-hosted inference the governing fact is hardware utilisation: “Inference hardware utilization is very important in terms of serving costs. GPUs are expensive and we need them to do as much work as possible.” That is the economics behind the batching described in the L2 sheets: continuous batching exists so that one load of the weights serves several requests.
For hosted APIs the bill is metered per token, and cached input is the largest predictable lever. The provider’s pricing documentation states the multipliers: cache writes cost “25% more than base input tokens for 5-minute TTL”, cache reads cost “10% of base input token price”, and a longer-lived cache — “a 1-hour TTL at 2x the base input token price” — trades a higher write cost for a longer window. Those are one provider’s numbers and its own policy; the shape — cheap reads, more expensive writes, a lifetime that decides whether you win — generalises.
Caching is a design decision, not a switch
The mechanism is described as caching a prefix: “Cache prompt prefixes with cache_control to cut costs and
latency, using automatic caching or explicit breakpoints with 5-minute or 1-hour TTLs.”
Three consequences follow for a retrieval pipeline:
- Order the prompt so the stable part comes first. Instructions, tool definitions and the corpus excerpts that do not change between turns belong before the volatile question; a prefix cache can only help if the prefix is genuinely identical.
- Short prompts do not benefit. “Shorter prompts cannot be cached, even if marked with
cache_control.” A pipeline that keeps its context small has little to cache. - The benefit is measurable per request. The API’s usage object reports the cached portion alongside the prompt and completion tokens, so the saving can be verified rather than assumed — the same discipline the L2 measurement sheet applies to any performance claim.
For a retrieval system specifically, caching is unusually profitable: the retrieved documents are the large, stable part of the prompt within a conversation, and the user’s question is the small changing part.
Latency is not cost, and both are not throughput
- A cached read is cheaper and faster — the two move together, which is not true in general.
- A reranker is slower and can be cheaper: it adds a step, and it lets you pass fewer chunks to the model (the reranking sheet quotes the source making exactly this point).
- Batching improves throughput and worsens per-user latency — the trade the L2 measurement sheet defines.
- Retrieval adds a floor to time-to-first-token that no model tuning removes: embed, search, (rerank), then prefill.
The practical discipline is the one the L2 batch already states for generation: report latency with its conditions — model, runtime, hardware, context and concurrency — and add, for this layer, the retrieval configuration and the number of chunks in the context.
Keeping it bounded
- Bound the request, not the month. Maximum output tokens, a maximum number of chunks, a maximum context size: each is a design decision with a quality consequence, and each is cheaper to enforce than to discover.
- Rate limits and quotas protect the system from its own clients — the L2 API sheet classifies them; the risk of not having them is OWASP’s LLM10, Unbounded Consumption, where excessive inference “expands on what was previously Denial of Service to include risks around resource management”.
- Attribute the cost. Per-request usage records let a team say which feature, which tenant or which query pattern consumes the budget — the difference between engineering and arguing about the invoice.
Level and prerequisites. L3 — integration level: the reader must be able to state what a request costs, in tokens, latency and money, and to choose the levers that bound it. No price list is quoted as current and no system is instrumented here. Prerequisites: the L2 measurement sheet, and the retrieval sheets of this layer that decide how many chunks there are to pay for.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Operations and risk — the node this sheet sits in.
References
- Databricks — LLM Inference Performance Engineering — hardware utilisation as the cost driver, and the latency metrics used by the L2 sheet.
- Anthropic — Prompt caching — the prefix-caching mechanism, the TTL options, the write/read multipliers, the minimum prompt requirement and the usage reporting.
- OpenAI — API description — the usage object’s cached-token field.
- OWASP — Top 10 for LLM Applications (2025), LLM10 — unbounded consumption as a resource risk.