An input token and an output token ask the model for the same arithmetic. On one H100 the first takes about 41 microseconds and the second about 5.8 milliseconds, both derived below. The factor of 142 has little to do with the arithmetic, and it is time on a card you own rather than money on an invoice. Rented inference bills a much smaller ratio.
A model answers one token at a time. It runs every layer over every earlier position, samples a token, appends it, then runs again. The prompt goes through once in one batch, which is compute-bound. Every answer token re-reads the weights and the whole key-value cache, which is memory-bound. The cache is what turns context length into money.
How a language model produces the next token
Tokenise the text, look every token up in an embedding table, run each layer’s attention over every earlier position, project the final position onto the vocabulary, sample one token and start again. Four stages, one of which repeats. Inside a layer every position becomes a query, a key and a value, mixed by how well queries match keys.
Attention splits that work across heads. Section 3.2.2 of Attention Is All You Need says the authors used “h=8 parallel attention layers”, and sets each head’s dimension at d_k = d_v = d_model/h = 64: model dimension over head count. Every number below runs through it.
The repeat is forced. “We need to prevent leftward information flow in the decoder to preserve the auto-regressive property,” says section 3.2.3 of the same paper. Token 401 can’t be computed until token 400 exists. Generation doesn’t parallelise inside one sequence, and no hardware fixes it — which is why what we build with applied AI gets costed in two halves.
What the KV cache is, and why it turns context length into money
Every token’s key and value vectors are kept once computed, so the next token doesn’t recompute them, and that store grows with the sequence while the weights don’t. Nobody calls it a cache because it’s optional. Throw it away and token 401 means recomputing keys and values for all 400 positions before it: quadratic work for linear output.
Section 3 of Efficient Memory Management for Large Language Model Serving with PagedAttention, the SOSP ’23 paper behind vLLM, prints it: “for the 13B parameter OPT model [62], the KV cache of a single token demands 800 KB of space, calculated as 2 (key and value vectors) x 5120 (hidden state size) x 40 (number of layers) x 2 (bytes per FP16).” Multiply it out. 2 x 5120 x 40 x 2 = 819,200 bytes, and 819,200 / 1,024 = 800 KiB exactly, so the paper’s “KB” means KiB.
Now the quirk, which takes under a minute to confirm. Open the arXiv HTML of that paper, find the sentence and copy it out. The leading factor arrives twice, as “calculated as 2 2 (key and value vectors)”, and so does every factor after it, because the markup carries a presentation node alongside a TeX annotation, and extraction takes both. A browser with MathML support shows one 2 and hides the other, so the page looks right and the copy is wrong. Park that. It comes back at the end.
The paper is blunt about where the shortage leads: from the A100 to the H100 the FLOPS “increases by more than 2x, but the GPU memory stays at 80GB maximum”, so “the memory will become an increasingly significant bottleneck.”
Working out the KV cache size for Llama 3.1 8B at 128K context
Multiply 2 by the layer count, the key/value head count, the head dimension, the bytes per element and the sequence length. Table 3 in section 3.2 of The Llama 3 Herd of Models gives the 8B column as 32 layers, model dimension 4,096, FFN dimension 14,336, 32 attention heads, 8 key/value heads and a vocabulary of 128,000. Head dimension is 4,096 / 32 = 128. The bullet above it is where the four-times error lives: the architecture uses “grouped query attention (GQA; Ainslie et al. (2023)) with 8 key-value heads to improve inference speed and to reduce the size of key-value caches during decoding”. Size the cache on 8 heads, not 32.
Section 4.3.4 says “we extend the context length of Llama 3 from 8K tokens to 128K tokens”, which is 128 x 1,024 = 131,072 positions. The paper never prints that expansion, so it’s mine. Everything else below is Table 3’s.
text
head_dim 4,096 model dim / 32 attention heads = 128
KV bytes/token 2 (K and V) x 32 layers x 8 KV heads x 128 x 2 (FP16) = 131,072 = 128 KiB = 0.125 MiB
KV at 128K ctx 0.125 MiB x 131,072 tokens = 16,384 MiB = 16 GiB
same in decimal 16 GiB x 1.073741824 = 17.18 GB
attn / layer Q 4,096x4,096 + K 4,096x1,024 + V 4,096x1,024 + O = 41,943,040
(K and V are 1,024 wide: 8 KV heads x 128 head_dim)
SwiGLU / layer 3 x 4,096 x 14,336 = 176,160,768
per layer = 218,103,808
x 32 layers = 6,979,321,856
+ embed, output 2 x 128,000 x 4,096 = 1,048,576,000
total params = 8,027,897,856
weights in FP16 8,027,897,856 x 2 bytes = 16,055,795,712 bytes = 16.06 GB = 14.95 GiBSet the cache equal to the weights and the crossover is 122,496 tokens: weights are 2P bytes, cache is 131,072n bytes, so n = P / 65,536 and 8,027,897,856 / 65,536 = 122,496. A 128K window sits just past that line, which is the second route to the fence above it: 16 GiB of cache against 14.95 GiB of weights. The cache is bigger than the model.
The 8,027,897,856 is where the 8B comes from, near enough: the sum takes Table 3’s rounded 128,000 vocabulary and drops the RMSNorm weights, so it sits a shade under the checkpoint. Reach for 32 attention heads instead of 8 key/value heads and you get 2 x 32 x 32 x 128 x 2 = 524,288 bytes, 512 KiB a token, a 64 GiB window. Four times too big — grouped query attention shares one key/value head across four query heads.
Why is an output token more expensive than an input token?
Because the prompt goes through the machine once as one wide matrix multiply, and each output token drags the whole model and the whole cache back across the memory bus. Take Lamplighter, an invented document-retrieval system over a contracts archive. Nobody’s archive is being described: every token count was picked so the divisions come out legible, and the prompt here runs longer than Lamplighter’s elsewhere because a short one never fills a cache. A query sends 24,200 input tokens — 6,000 of fixed instruction and schema, 18,000 of retrieved contract text, 200 of question — and gets 400 back.
NVIDIA’s H100 product page lists FP16 Tensor Core at 1,979 teraFLOPS with an asterisk; the footnote under the table reads “With sparsity.” A dense forward pass has no 2:1 structured sparsity, so the honest figure is 1,979 / 2 = 989.5 TFLOPS, derived here because the table prints no dense figure. Assume the kernels reach 40 per cent of it, the one genuinely chosen number on this page. Memory bandwidth on the same table is 3.35 TB/s. A matrix multiply does one multiply and one add per weight, so call a forward pass 2 FLOPs per parameter per token.
text
prefill 2 x 8,027,897,856 params x 24,200 tokens = 388.55 TFLOP
989.5 TFLOPS dense x 0.40 chosen utilisation = 395.8 TFLOPS
388.55 / 395.8 = 0.982 s
decode mean context 24,200 + 400/2 = 24,400 tokens
16.06 GB weights + 131,072 x 24,400 bytes = 19.25 GB per output token
400 x 19.25 GB / 3,350 GB/s = 2.299 sBut about 2.3 seconds to write 400 tokens against about 1.0 to read 24,200 is a wall-clock ratio of only 2.3x, and per token it is 0.982 / 24,200 = 40.6 microseconds against 2.299 / 400 = 5.75 milliseconds: 5,750 / 40.6 = 142. Sixty and a half times as many tokens went through prefill, and prefill still finished in under half the time.
Prefill is compute-bound, decode is memory-bandwidth-bound, and the ridge is 295 FLOP per byte
Prefill has N tokens of arithmetic against one pass over the weights; decode has one token of arithmetic against the same pass. Divide a card’s arithmetic rate by its memory bandwidth and you get its ridge point: the arithmetic intensity below which it starves for data. Below the ridge, the card waits. On the H100 SXM the ridge is 989.5 TFLOPS of derived dense FP16 over 3.35 TB/s, so 989.5 / 3.35 = 295.4 FLOP per byte.
A decode step does about 2 FLOPs per parameter and reads 2 bytes per parameter: roughly 1 FLOP per byte. 295 times short of the ridge. Prefill does N times the arithmetic over the same bytes, so it crosses at around 295 prompt tokens. Anything longer than a short paragraph is already compute-bound, and nothing you generate ever is.
Memory bandwidth is the decode budget, so count bytes. An output token with an empty cache reads 16.06 GB of weights: at 3,350 GB/s, 208.6 tokens a second, 4.79 ms each. With the 128K window full it reads 16.06 + 17.18 = 33.24 GB, so 100.8 a second and 9.92 ms. Filling the window costs 33.24 / 16.06 = 2.07x on every token after it. The model didn’t get slower. The bus got busier.
Capacity works the same way. vLLM’s engine arguments page gives --gpu-memory-utilization a default of 0.92, “The fraction of GPU memory to be used for the model executor” — an odd number to default to, and the 8 per cent it withholds is why the arithmetic lands on 3.35 sequences rather than (80 - 16.06) / 17.18 = 3.7. On an 80 GB card the executor gets 73.6 GB, weights take 16.06 GB, and 57.54 / 17.18 = 3.35 full-length sequences. Three and a bit, on a card rated at up to 700 W. Serve 32K instead (32 x 1,024 = 32,768 tokens) and the cache falls to 131,072 x 32,768 = 4.295 GB a sequence: 57.54 / 4.295 = 13.4 fit on a card. Refusing the full window is the cheapest capacity lever here.
What prompt caching saves, and what it leaves untouched
Prompt caching skips the prefill arithmetic for a prefix the server has already seen, so it buys back part of the compute-bound half of a request and none of the memory-bound half. Anthropic’s prompt caching documentation prices it in three lines under the pricing table: a 5-minute write at 1.25 times base input, a 1-hour write at 2 times, and “Cache read tokens are 0.1 times the base input tokens price (see the table footnote for per-model exceptions)”. The parenthesis matters: every figure in the next section rests on that 0.1, taken as given for the models I’d put a retrieval system on. Without a cache, n requests over one prefix cost n; with the five-minute cache, 1.25 + (n - 1) x 0.1. That is 0.9n = 1.15, so n = 1.278: the write pays back on the second hit, and the 2.0 write needs a third.
The fine print on that page joins this piece’s two halves. The “How prompt caching works” section says: “The lifetime is measured from the start of the request that writes or reads the cache entry, not from the end of its response. Time spent generating a response counts against the lifetime: if a response takes 4 minutes to stream, a follow-up request that reuses the same cached prefix must start within about 1 minute of that response completing.” Slow decode spends the cache window it was meant to benefit from: four of the default five minutes, gone to the bus. OpenAI’s prompt caching guide sets a different floor for the same mechanism, 1,024 tokens for GPT-5.6 and later, and what either stack does underneath is unpublished.
When prompt caching is worth it: the stable-prefix share
Below a 40 per cent stable-prefix share we don’t wire prompt caching into a retrieval system. The stable-prefix share is the fraction of a request’s input tokens byte-identical to the previous request’s, measured at the last cache breakpoint. Where the 40 comes from: a third of the input bill is the point below which I stop arguing for an extra moving part, cache reads bill at 0.1 so the ceiling is the stable share x 0.9, and a third back needs 0.333 / 0.9 = 37 per cent. Round up to 40.
Lamplighter is under the line. Its fixed instruction-and-schema block is 6,000 of 24,200 input tokens and the retrieved text changes every query, so price the bill and the clock separately.
text
stable-prefix share 6,000 fixed / 24,200 input tokens = 24.8 %
ceiling 24.8 % x (1 - 0.1 cache read) = 22.3 % of the input bill
cached input cost 6,000 x 0.1 + 18,200 x 1.0 = 18,800 units against 24,200
saving 5,400 / 24,200 = 22.3 %
cached prefill 2 x 8,027,897,856 x 18,200 / 395.8e12 = 0.738 s, down from 0.982
latency saving (0.982 - 0.738) / (0.982 + 2.299) = 7.4 % of the wall clockThe ceiling and the saving agree at 22.3 per cent, which proves nothing: share x 0.9 is the cost division rearranged. On the clock the cache buys 7.4 per cent, because caching touches prefill and prefill was never the slow half.
For a retrieval system, prompt caching is mostly theatre.
Retrieved passages are the bulk of the prompt and differ by construction on every query, so the only cacheable part is the instruction block — the argument semantic caching and the repeat rate makes at length, borrowed here rather than discovered. There’s an honest case against it, and I’d make it myself: if several questions in a session pull the same contract, the retrieved block is stable too and caching wins. But cost it. At a 100 per cent share the ceiling is 100 x 0.9 = 90 per cent of the bill, and we would build it. Lamplighter doesn’t: its retrieval runs per question. Attack the retrieved context instead, the argument in retrieval before fine-tuning. Double the whole input to 48,400 tokens and the share halves to 6,000 / 48,400 = 12.4 per cent, the ceiling to 12.4 x 0.9 = 11.2 per cent. More context makes caching worse, not better.
Check the model FLOPs utilisation before trusting this KV cache arithmetic
Model FLOPs utilisation at 40 per cent is the only number on this page nobody published, and I picked it myself. Derate it fourfold to 10 per cent and prefill becomes 388.55 / 98.95 = 3.927 seconds against the same 2.299 of decode, so the wall-clock conclusion flips. Below roughly 17.1 per cent utilisation — 388.55 / (989.5 x 2.299) — prefill is the longer stage, and an 8B model at batch size 1 on a short prompt can sit there; above it decode wins, by 2.3x at 40 per cent. The per-token result survives the derating: 142x becomes 35.4x.
Two things I’d check rather than trust. Extract that doubled factor as 22 and you get 9,011,200 bytes, which doesn’t reconstruct PagedAttention’s own 800 KB. The PDF says 2, and I opened it only because the arithmetic refused to close. I also don’t know vLLM’s current default block size and wouldn’t quote one from memory: section 7.2 of that paper says “vLLM sets its default block size as 16”, and the engine arguments page prints no static default.
Which number you optimise is set by how you pay, not by the model. Rent it and input is 92.4 per cent of the bill: 24,200 input units against 400 x 5 = 2,000 output units at a five-to-one price ratio I chose rather than quoted, 26,200 in all. Run it yourself and the lever is the cache, at 3.35 sequences a card.