Articles

Concurrent execution and resource management

Batching, queueing and backpressure: running many model calls within one pool.

Reading: 4 minAutomation & AI

Article cover: Concurrent execution and resource management

Running many model calls at once is not a switch you turn up: it is a claim on a pool of accelerator memory and compute that every in-flight call shares. Each request holds a key-value cache that grows with its prompt and output, and the pool — GPU memory, memory bandwidth, the tokens the scheduler pushes through each iteration — is finite. The engineering is deciding how much of that pool one caller may hold, what happens when it is full, and how you notice the limit before it is crossed. The failure here is not a crash but a service that accepts work until every request is slower and the extra parallelism buys nothing.

Concurrency is a claim on a shared pool

Every in-flight request consumes accelerator memory whether or not it is being computed on. When a serving system manages that memory badly, “memory can be significantly wasted by fragmentation and redundant duplication, limiting the batch size” (vLLM paper, lines 19–20); a profiling of earlier systems found that “only 20.4% - 38.2% of the KV cache memory is used to store the actual token states” (lines 164–165). The calls a device can run at once are therefore set by memory efficiency, not by a value an operator picks: two machines with the same model admit different numbers depending on prompt and output length. How that cache is paged is serving mechanics; here it is the budget every concurrent call draws on.

Batching: what it buys, and what it costs

Batching shares one pool across many calls — static, dynamic, or at the iteration level: “Continuous batching: The idea of batching requests together as they arrive … is currently the SOTA method. Instead of waiting for all sequences in a batch to finish, it groups sequences together at the iteration level. It can achieve 10x-20x better throughput than dynamic batching” (Databricks, line 422). The gain is not free per request: “if we process 16 user queries concurrently, we’ll have higher throughput compared to running the queries sequentially, but we’ll take longer to generate output tokens for each user” (line 323). Throughput and per-caller latency move in opposite directions; choosing a point on that curve is the cost and performance sheet’s subject.

Saturation: when more concurrency stops helping

Batching does not scale forever. “After a certain batch size, i.e., when we cross to the compute bound regime, every doubling of batch size just increases the latency without increasing throughput” (Databricks, line 489) — a working definition of saturation: the hardware is already busy, so extra concurrent work is queued rather than executed. Time to first token, the metric for “users start seeing the model’s output after entering their query” (line 313), includes the wait before a request is scheduled, so latency, not throughput, is the first sign that the pool is full.

When the limit is reached: admission control and backpressure

Two controls keep a service inside its limit. The server can refuse work with a status the client understands: “The 429 status code indicates that the user has sent too many requests in a given amount of time” (RFC 6585, lines 134–135), optionally with a “Retry-After header indicating how long to wait before making a new request” (lines 138–139). Refusing is itself costly — “responding to each with a 429 status code will consume resources” — so the same document notes that when “limiting resource usage, it may be more appropriate to just drop connections” (lines 297–298, 301–302). The client’s lever is to bound its own concurrency: “If errors are caused by load, retries can be ineffective if all clients retry at the same time”, which is why “backoff and jitter can improve utilization and reduce congestion on systems” (AWS, lines 84, 142).

What you measure to see it coming

A serving engine exposes the pool directly: server-level metrics, “Global metrics that track the state and performance of the LLM engine”, that “explain the values of request-level metrics” (vLLM metrics, lines 2647, 2651). The signals of a pool filling up are the request states — running as well as waiting (lines 2657, 2661) — the cache in use, gpu_cache_usage_perc (line 2663), the queue-time histogram (line 2755), and preemptions, where “the request has been put back in the waiting queue in order to make room for other requests to complete” (line 2886). Rising waiting and cache usage together mean the pool is nearly full.

Failure modes this design invites

  • Coordinated bursts. Clients ticking on the same schedule line up, and the pool absorbs the spike at once; spreading arrivals is cheaper than adding capacity.
  • Retry amplification under load. Every caller retrying at the same moment adds load instead of relieving it, so the limit belongs at one level of the stack.
  • Preemption churn. Past the limit the scheduler evicts and re-runs requests instead of finishing them.

Level and prerequisites. L5 — the decision is a resource budget for a running system: how much concurrency a pool can serve, what to do when demand exceeds it, and which signals prove it is about to. Prerequisites: the earlier sheets of this area on how a model is served and what the key-value cache is, and the L4 sheets on retries and loop control.

Where to go next

References

  • Kwon et al. — Efficient Memory Management for Large Language Model Serving with PagedAttention — the memory waste and cache-efficiency figures that bound a batch.
  • Databricks — LLM Inference Performance Engineering: Best Practices — the batching techniques, the throughput-versus-latency effect of concurrency, and the compute-bound saturation point.
  • RFC 6585 — Additional HTTP Status Codes — the 429 status code, Retry-After, and the cost of refusing each request.
  • AWS Builder Center — Timeouts, retries, and backoff with jitter — bursts, jitter, and the effect of arrival shape on a shared pool.
  • vLLM documentation — Metrics — server-level versus request-level metrics and the states that signal saturation.
Nodrius Field Kit — cover

Get the Nodrius Field Kit

Enter your email and we'll send you the complete Field Kit: eight technical resources in one download.

Your email address is sent to Nodrius and used to deliver the Field Kit to you by email. Marketing messages are separate and optional: the checkbox does not affect your download, and you are only added to our updates list if you tick it. Privacy Policy.

Privacy & cookies

This site does not use cookies, analytics or tracking, and it does not profile you. There is nothing technical to switch off, so Accept and Reject change nothing about how the site works: either choice lets you browse everything normally.

The only difference is what your browser remembers: two entries in local storage — your choice here, so this notice is not shown again, and a marker that the Field Kit offer has already been made — so the offer is presented once and not on every page. They contain no identifier, they are not shared with anyone, and the site sets no cookie for either.

What we do with your data
How your email address is used when you ask for a Resource, how long it is kept and which rights you have — in the Privacy Policy.
What the site stores in your browser
Two entries in local storage — your choice on this notice and whether the Field Kit offer has been shown — and no cookies, no session storage, no third-party content: in the Cookie Policy.