The idea: one forward pass serves the whole batch
A language model writes text one token at a time, and each token needs a pass through every layer of the model. Reading all the weights from GPU memory is the slow part, so a server runs many requests in the same pass: a batch. Each request in the batch has a slot, and in every step (one forward pass) every busy slot writes one token. The question is when a request may take a slot and when it must give it back.
The canvas runs the same 8 requests through two servers with 4 slots each, one step at a time on a shared
clock. Every request is waiting at step 0 and writes a fixed number of tokens (R1·8 writes 8).
Each cell of a lane's grid is one slot in one step.
Prefill and decode
A request's first step is its prefill: the model reads the whole prompt in one pass,
fills the KV cache and writes the first token (P on the canvas).
Every later step is a decode step that writes one more token. The time from arrival to
the first token is the time to first token (TTFT); the speed after that is the time per
output token. On the canvas every step costs one tick; real prefill steps cost more for long prompts.
Static batching: waiting for the longest
With static batching the server takes 4 requests, runs them together until the longest one is done, then takes the next 4. A short request that is done after 2 steps keeps its slot (as padding) while the long one goes on, and the requests in the queue cannot start. In the mixed workload R1 writes 8 tokens, so R2, R3 and R4 leave three slots idle for most of the first batch, and R5 to R8 get their first token only at step 9. (Demo: first slots free — static idles, continuous refills)
Continuous batching: join and leave every step
Continuous batching (also called in-flight or iteration-level batching, first described in the Orca paper and used by vLLM, TGI and TensorRT-LLM) makes the decision after every step instead of after every batch. A request that has written its last token leaves at once, and a waiting request takes its slot in the next step, doing its prefill while the others decode. The batch never waits for its longest member. (Demo: mixed lengths to the end — 15 vs 9 steps)
The numbers
Both servers write exactly the same tokens; only the number of steps differs. Throughput is tokens per step (at 25 ms per step, multiply by 40 for tokens per second), and slot use is tokens divided by 4 × steps.
| workload | steps static / continuous | tokens/step | slot use | mean TTFT (steps) |
|---|---|---|---|---|
| mixed lengths: 8 2 3 2 7 2 3 2 | 15 / 9 | 1.93 / 3.22 | 48% / 81% | 5.0 / 2.4 |
| all the same length: 4 × 8 | 8 / 8 | 4.00 / 4.00 | 100% / 100% | 3.0 / 3.0 |
| one long request: 12 then 2 × 7 | 14 / 12 | 1.86 / 2.17 | 46% / 54% | 7.0 / 2.3 |
When every request has the same length, nothing finishes early, so the two are the same (Demo: same lengths — no difference). Real replies vary from one token to thousands, which is why every modern LLM server batches continuously. Continuous batching cannot make a single request faster: with one 12-token request the run still takes 12 steps, and the last steps have one busy slot because the queue is empty. But the short requests no longer wait behind it, so their TTFT drops from 13 steps to 3–5 (Demo: one long request — 14 vs 12 steps). That is the trade-off of latency vs throughput: a full batch gives throughput, and a slot that frees early gives the next request its latency back.
What limits the batch
A slot is not free: each running request holds its keys and values in the KV cache, so GPU memory sets how many requests fit. Continuous batching needs memory that can be handed out a block at a time, which is what PagedAttention provides. A long prompt joining the batch makes that step slow for everyone, so servers split it into pieces (chunked prefill) and mix them with the decode steps. Speculative decoding attacks the same problem from the other side: more tokens per step for one request instead of more requests per step.
See also How a Large Language Model Works for the token-by-token loop and Self-Attention for what each step computes.
How many requests fit in a batch depends on the KV-cache size per token, which grouped-query attention cuts by 8 times on Llama 3 70B.
What the page leaves out
Real step costs (prefill time grows with the prompt, decode time with the batch and the cache), requests arriving over time, prompt lengths, chunked prefill, preemption and swapping when the KV cache is full, priorities and fairness, a maximum token budget per step, stop tokens that end a reply early, and multi-GPU tensor or pipeline parallelism.