Cost, performance and architectural trade-offs
Model, cache and context choices set unit cost — and a measured number does not transfer.

The cost of an AI service is not a property of the model but of a configuration — model, hardware, batch policy, cache and context length. Move one lever and the price and the latency of the whole move with it. Price is paid per token and per second of hardware, latency per user: raising throughput by making users wait longer trades one currency for another rather than removing the cost.
A cost is a configuration, not a model
Decoding is memory-bound: the machine spends its time loading model parameters, not computing on them. Databricks draws the result — “Available and achieved memory bandwidth in inference hardware is a better predictor of speed of token generation than their peak compute performance” (line 351). The number belongs to the model, the hardware and the software stack together — which is why the PagedAttention paper frames the “cost per request” of serving systems as a systems problem, not a model property (line 45).
The dimensions a measurement lives on
The serving vocabulary has four axes. Time To First Token (TTFT) is the wait before output starts; Time Per Output Token (TPOT) its pace afterwards; latency is the total — “latency = (TTFT) + (TPOT) * (the number of tokens to be generated)”; throughput is the rate across all users (Databricks, lines 313, 315, 317, 319). Two heuristics bound the design: “Output length dominates overall response latency” (line 327), while “Input length is not significant for performance but important for hardware requirements” (line 329) — a long prompt costs memory, not much waiting.
Throughput and latency trade against each other
Batching is the main lever on unit cost, and it is not free: concurrency buys aggregate throughput by making each user wait longer per token (Databricks, line 323). On one A100, “if we maximize throughput with a batch size of 64, latency increases by 4x while throughput increases by 14x” (line 487); and past a certain size “every doubling of batch size just increases the latency without increasing throughput” (line 489).
The limit is memory: a larger batch needs a larger KV cache, and that needs more hardware — “a large batch means larger KV cache size, and that in turn increases the number of GPUs required to serve the model” (line 353). PagedAttention’s claim is that better memory management removes that waste rather than passing it on, reaching “near-zero waste in KV cache memory” (line 121). Running many calls at once is the concurrent-execution sheet’s subject.
A cache moves the unit economics
Caching changes what a repeated request pays for: a prompt cache stores a prefix so a later request resumes from it and reuses the expensive part (Anthropic, line 92). The accounting is asymmetric: writing a prefix to the cache costs more than processing it fresh, while reading it back costs far less (lines 416, 585). A hit needs “100% identical prompt segments” (line 786), so the saving is bounded by how exactly traffic repeats.
Serving-side prefix caching has the same shape — vLLM reuses “the KV cache of existing queries … allowing the new query to skip the computation of the shared part” (line 458). But it “only reduces the time of processing the queries (the prefilling phase)” and not decoding (line 482): a cache is a bet on repetition, and where there is none it pays its own overhead for nothing.
A cheaper model is not a cheaper system
Model choice is an economic decision before it is a routing decision. RouteLLM routes between a strong model giving “high-quality responses at a high cost” and a weak one giving “lower-quality responses at a reduced cost”, to “minimize costs while achieving a specific performance target, e.g., 90% of the strong model” (line 24); its call-performance threshold — “the minimum percentage of calls to the strong model needed to reach the desired PGR” (line 159) — prices a quality target rather than a rate. The saving is bounded by quality: a cheaper model that triggers more retries has moved the cost, not removed it. Where a request goes is the routing sheet’s question; what it costs is this sheet’s.
Why a measured number does not transfer
A benchmark is only as good as the configuration it was measured on. Two clouds can run the same GPU count at different speeds: Databricks observed “a 2x latency difference between 8xA100 servers from two cloud providers” (line 549). Token-based figures are not comparable across model families, because tokenisation differs — “these numbers are not always comparable across model types given these variations” (line 305). The rule follows: “we recommend always measuring end-to-end server performance” (line 549), under the configuration you will actually run.
Level and prerequisites. L5 — the systems-engineering level: the reader must treat cost and latency as properties of a configuration, name the dimension a decision is measured on, and separate a saving that removes work from one that relocates it. Prerequisites: the L1 and L2 sheets of this area (how a model generates text, its context window, how it is served), the L3 and L4 sheets on retrieval, tool calling and agent loops, and the fallback and concurrency sheets.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Applications — the node this sheet sits in.
References
- Ong et al. — RouteLLM: Learning to Route LLMs with Preference Data — the strong/weak formulation, the cost objective, and the call-performance threshold as a cost metric.
- Databricks (MosaicML) — LLM Inference Performance Engineering: Best Practices — the four serving metrics, the throughput/latency trade-off, the memory-bandwidth bottleneck, the KV-cache tug-of-war, and why a measured number does not transfer.
- Anthropic — Prompt caching — the write/read cost asymmetry of a cached prefix, the exact-match constraint, and the latency effect for long documents.
- vLLM — Automatic Prefix Caching — prefix reuse, and its limit to the prefill phase.
- Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention — the systems framing of serving cost and near-zero KV-cache waste.
- vLLM — Metrics — request-level metrics as the tracked SLOs, server-level metrics explaining them.