Concurrent execution and resource management
Batching, queueing and backpressure: running many model calls within one pool.

Running many model calls at once is not a switch you turn up: it is a claim on a pool of accelerator memory and compute that every in-flight call shares. Each request holds a key-value cache that grows with its prompt and output, and the pool — GPU memory, memory bandwidth, the tokens the scheduler pushes through each iteration — is finite. The engineering is deciding how much of that pool one caller may hold, what happens when it is full, and how you notice the limit before it is crossed. The failure here is not a crash but a service that accepts work until every request is slower and the extra parallelism buys nothing.
Concurrency is a claim on a shared pool
Every in-flight request consumes accelerator memory whether or not it is being computed on. When a serving system manages that memory badly, “memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size” (vLLM paper, lines 19–20); a profiling of earlier systems found that “only 20.4% - 38.2% of the KV cache memory is used to store the actual token states” (lines 164–165). The calls a device can run at once are therefore set by memory efficiency, not by a value an operator picks: two machines with the same model admit different numbers depending on prompt and output length. How that cache is paged is serving mechanics; here it is the budget every concurrent call draws on.
Batching: what it buys, and what it costs
Batching shares one pool across many calls — static, dynamic, or at the iteration level: “Continuous batching: The idea of batching requests together as they arrive … is currently the SOTA method. Instead of waiting for all sequences in a batch to finish, it groups sequences together at the iteration level. It can achieve 10x-20x better throughput than dynamic batching” (Databricks, line 422). The gain is not free per request: “if we process 16 user queries concurrently, we’ll have higher throughput compared to running the queries sequentially, but we’ll take longer to generate output tokens for each user” (line 323). Throughput and per-caller latency move in opposite directions; choosing a point on that curve is the cost and performance sheet’s subject.
Saturation: when more concurrency stops helping
Batching does not scale forever. “After a certain batch size, i.e., when we cross to the compute bound regime, every doubling of batch size just increases the latency without increasing throughput” (Databricks, line 489) — a working definition of saturation: the hardware is already busy, so extra concurrent work is queued rather than executed. Time to first token, the metric for “users start seeing the model’s output after entering their query” (line 313), includes the wait before a request is scheduled, so latency, not throughput, is the first sign that the pool is full.
When the limit is reached: admission control and backpressure
Two controls keep a service inside its limit. The server can refuse work with a status the client understands: “The 429 status code indicates that the user has sent too many requests in a given amount of time” (RFC 6585, lines 134–135), optionally with a “Retry-After header indicating how long to wait before making a new request” (lines 138–139). Refusing is itself costly — “responding to each with a 429 status code will consume resources” — so the same document notes that when “limiting resource usage, it may be more appropriate to just drop connections” (lines 297–298, 301–302). The client’s lever is to bound its own concurrency: “If errors are caused by load, retries can be ineffective if all clients retry at the same time”, which is why “backoff and jitter can improve utilization and reduce congestion on systems” (AWS, lines 84, 142).
What you measure to see it coming
A serving engine exposes the pool directly: server-level metrics, “Global metrics that track the state
and performance of the LLM engine”, that “explain the values of request-level metrics” (vLLM metrics,
lines 2647, 2651). The signals of a pool filling up are the request states — running as well as waiting
(lines 2657, 2661) — the cache in use, gpu_cache_usage_perc (line 2663), the queue-time histogram (line
2755), and preemptions, where “the request has been put back in the waiting queue in order to make room
for other requests to complete” (line 2886). Rising waiting and cache usage together mean the pool is
nearly full.
Failure modes this design invites
- Coordinated bursts. Clients ticking on the same schedule line up, and the pool absorbs the spike at once; spreading arrivals is cheaper than adding capacity.
- Retry amplification under load. Every caller retrying at the same moment adds load instead of relieving it, so the limit belongs at one level of the stack.
- Preemption churn. Past the limit the scheduler evicts and re-runs requests instead of finishing them.
Level and prerequisites. L5 — the decision is a resource budget for a running system: how much concurrency a pool can serve, what to do when demand exceeds it, and which signals prove it is about to. Prerequisites: the earlier sheets of this area on how a model is served and what the key-value cache is, and the L4 sheets on retries and loop control.
Where to go next
- Automation & AI — the macro-area this sheet belongs to.
- Applications — the node this sheet sits in.
References
- Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention — the memory waste and cache-efficiency figures that bound a batch.
- Databricks — LLM Inference Performance Engineering: Best Practices — the batching techniques, the throughput-versus-latency effect of concurrency, and the compute-bound saturation point.
- RFC 6585 — Additional HTTP Status Codes — the 429 status code,
Retry-After, and the cost of refusing each request. - AWS Builder Center — Timeouts, retries, and backoff with jitter — bursts, jitter, and the effect of arrival shape on a shared pool.
- vLLM documentation — Metrics — server-level versus request-level metrics and the states that signal saturation.