The idea: old keys and values never change
A language model writes one token at a time. To pick the next token, the newest token's query is compared with the key of every earlier token, and their values are mixed (see Self-Attention). In a decoder the causal mask means a token never looks ahead, so the key and value of token 3 are the same at step 4, step 5 and step 100. The KV cache keeps them: each step computes K and V for the new token only and appends them.
The canvas follows one request. The prompt is “the cat sat” and the model writes “on the soft mat .” (the reply is scripted: the page is about cost, not about which token wins). The left lane has no cache, the right lane has one. Under them, the memory the cache takes and how a server stores it in blocks.
Without vs with the cache
The first pass is the prefill: K and V for all 3 prompt tokens in one go. Both lanes do the same work here. Each later press is a decode step: the newest token goes in, gets its q, k and v, attends over all K,V rows and predicts the next token. (Demo: prefill, then one decode step)
| step | token fed | K,V rows without cache | K,V rows with cache |
|---|---|---|---|
| prefill | the cat sat | 3 | 3 |
| decode 1 | on | 4 | 1 |
| decode 2 | the | 5 | 1 |
| decode 3 | soft | 6 | 1 |
| decode 4 | mat | 7 | 1 |
| total | 25 | 7 |
Without a cache, writing n tokens computes about n²/2 K,V rows, in every layer; with it, n. The attention itself (q against every key) is the same in both lanes and gives the same numbers, so the cache changes the cost, never the text. (Demo: the whole reply — 25 vs 7 K,V rows) How a Large Language Model Works shows the same loop on a real tiny model.
How much memory
The price is memory. Per token, the cache stores a key and a value for every KV head in every layer:
bytes per token = 2 × n_layers × n_kv_heads × d_head × bytes per number| model (fp16) | layers | KV heads × d_head | per token | 4,096 tokens |
|---|---|---|---|---|
| GPT-2 small | 12 | 12 × 64 | 36 KiB | 144 MiB |
| Llama-2-7B | 32 | 32 × 128 | 512 KiB | 2 GiB |
| Llama-3-8B | 32 | 8 × 128 | 128 KiB | 512 MiB |
| Llama-2-70B | 80 | 8 × 128 | 320 KiB | 1.3 GiB |
A server runs many requests at once, so 16 requests of 4,096 tokens on Llama-2-7B need 32 GiB of cache, more than the model’s own weights (about 13 GB). Grouped-query attention (GQA) lets several query heads share one K,V head, so Llama-3-8B stores 8 KV heads instead of 32: 4 times less. Multi-query attention goes to a single KV head. (Demo: GQA — Llama-3-8B, Demo: Llama-2-70B)
PagedAttention: the cache in blocks
A server does not know how long a reply will be. Reserving max_len slots per request up front
wastes most of them. PagedAttention (vLLM) splits the cache into blocks of a
fixed number of tokens (16 in vLLM, 4 on the canvas) from one shared pool. Each request has a block
table, a list of block numbers in order, so its blocks can sit anywhere in GPU memory, like pages in
virtual memory. A request takes a new block only when its last one is full, so
it wastes at most one partly filled block. (Demo: two requests share the block pool)
Blocks also make sharing cheap: two requests with the same system prompt can point at the same blocks for that prefix, and a block is copied only when one of them writes to it.
Prefill and decode feel different
Prefill handles all prompt tokens at once, so the GPU is busy with arithmetic (compute-bound); that is the wait before the first token. A decode step does little arithmetic for one token but reads the whole cache and all weights from memory (memory-bound); that is why later tokens stream at a steady rate, and why it gets slower as the cache grows. Batching many requests helps decode, and the cache is what limits the batch size.
See also How a Large Language Model Works for the whole generation loop, Self-Attention for what q, K and V are, Decoding Strategies for how the next token is chosen, and LoRA and Quantization for storing numbers in fewer bits (the cache can be quantized too).
What the page leaves out
Real tensors (each K,V row here is d_head numbers per head), multi-head detail, batching and scheduling of many requests, prefix caching and copy-on-write, sliding-window attention, KV cache quantization (8-bit or 4-bit), eviction when memory runs out, speculative decoding, and FlashAttention kernels.