Decode is memory-bound by construction

Prefill and decode run the same weights through the same kernels. They land on opposite sides of the roofline anyway — about 430× apart in arithmetic intensity on the configuration I worked through below.

That much is folklore. What I wanted was the closed form: not "decode is memory-bound," but how far from the ridge point, and what exactly you would have to change to move it. The answer turns out to be a two-line expression, and it says something fairly blunt — for multi-head attention at a short context, no batch size whatsoever gets decode to the ridge point. It is not a tuning problem.

The frame, compressed

A kernel doing $F$ FLOPs and moving $B$ bytes has two independent lower bounds on its runtime: $F/\text{peak FLOP/s}$ and $B/\text{peak bytes/s}$. On a modern GPU the two overlap — the next tile streams in while the current one multiplies — so the runtime is bounded by the larger, not the sum. Which one wins depends on one ratio, the arithmetic intensity $\mathrm{AI} = F/B$, compared against a property of the hardware alone:

H100 SXM: 989 TFLOP/s dense BF16 over 3.35 TB/s of HBM gives 295 FLOPs/Byte. Worth noticing that H200 keeps the same compute and adds bandwidth, which pulls the ridge point down to 206. For a memory-bound workload a lower ridge point is the improvement — that is the whole reason it is the better decode card.

The measurement

Reference model, chosen to be boring: $d = 4096$, 32 layers, 32 heads, head_dim 128, FFN $4d$, MHA, FP16. That is $12d^2$ parameters per layer, 6.44 B total, 12.88 GB of weights.

The compute fell by 512×. The memory traffic barely moved. Decode reads 13.2 GB out of HBM to produce one token, and the weight term — 12.88 GB of it — does not care how many tokens you are serving. Reading the weights is the cover charge for a forward pass.

Two regimes, not one

Averaging decode into a single AI number hides the mechanism. Decode is really two workloads with different scaling behaviour, sharing a kernel launch. Separating them is where it gets interesting.

The GEMM half. Weights are shared across the batch — read once, used by all $B$ sequences. So

Decode arithmetic intensity equals the batch size. At $B = 1$ that is AI = 1, matching the table. To reach 295 you would need ~295 concurrent sequences. This is the entire justification for continuous batching, and it is a clean one.

The attention half. Per layer per sequence, decode attention costs $4Sd$ FLOPs and reads $4Sd_{kv}$ bytes of KV cache. So

Everything cancels. The context length $S$ cancels. The batch size $B$ never appears at all — each sequence owns a private KV cache, so there is nothing to share across the batch.

For MHA, decode attention has an arithmetic intensity of 1. Not "1 at batch 1" — 1 at every batch size and every context length. No amount of batching moves it.

The only lever on that term is $g$: give several query heads one shared KV head. Which is exactly what GQA and MQA do. I had always filed those under "saves memory"; the ratio above says the bandwidth story is the same story, seen from the other side.

The ceiling

Put the halves together and let the batch run away:

The $B$ in the numerator cancels against the $B$ in the KV term, so this converges rather than growing. Substituting the standard shape $P = 12Ld^2$ collapses it to something you can hold in your head:

At $d = 4096$, $S = 512$, MHA: 49. Against a ridge point of 295. I checked this two ways — direct computation and the closed form agree to the decimal — because a result this convenient is usually a dropped factor of two.

Two things fall out of the $1/S$ dependence. At 2K context the ceiling is 13; at 8K it is 4.0; at 128K it is 1.2. And at 128K the KV cache for a single sequence is 68.7 GB against 12.9 GB of weights — the workload has stopped being about the model at all. It is a memory-streaming problem wearing a Transformer costume.

What continuous batching is actually buying

If $\mathrm{AI} \approx B$, then scheduling is a performance feature, not a plumbing detail. Two things stop you from getting the $B$ you configured.

Slot occupancy. A static batch runs until its slowest member finishes, so it burns $B \cdot \max_i L_i$ slot-steps while only $B \cdot \text{mean}_i L_i$ are useful. For iid exponential output lengths $\mathbb{E}[\max] = \text{mean} \cdot H_B$, so occupancy is exactly $1/H_B$ — decaying like $1/\ln B$. At $B = 32$ that is 24.6%: you configured 32 and you are running about 8. Continuous batching refills a slot the step after it frees, which is why it is worth the scheduler complexity.

Capacity. After weights, an 80 GB card has ~65 GB for KV cache. At 512 context that is 242 concurrent sequences; at 8K it is 15; at 32K it is 3. Note that capacity and intensity fail together — a longer context both shrinks the batch that fits and lowers the ceiling that batch could have reached. Two bad things, same cause.

What chunked prefill is actually buying

Different problem, same arithmetic. A prefill occupying the GPU stalls every decode step queued behind it, so users mid-generation watch their inter-token latency spike. At batch 32 an undisturbed decode step is ~8 ms. A 512-token prefill adds ~14 ms — survivable. An 8K prefill adds 285 ms, and a 32K prefill adds two seconds, because attention is quadratic and prefill is where that actually bites.

Split the prompt into chunks and interleave them with decode steps, and a queued decode waits for at most one chunk — a constant, independent of prompt length. At 32K that is 39 ms instead of 2000 ms.

So make the chunks small? No — and this is the part I like. A step processing $N$ tokens has $\mathrm{AI} \approx N$, by the same algebra that gave decode $\mathrm{AI} \approx B$. A chunk below the ridge point is memory-bound, wasting compute in precisely the way batch-1 decode does. The chunk size is squeezed from both ends: large enough to stay compute-bound, small enough that one chunk fits the TPOT budget. On H100 the lower end is ~295 tokens in theory and closer to 512 once activation traffic is counted.

The part I got wrong first

The tidy version of this is: tokens per step sets the AI, so continuous batching and chunked prefill are one mechanism seen from two ends — both exist to keep tokens-per-step above the ridge point.

That is a good sentence and it is not quite true. I built the table before I believed it, which is the only reason I noticed. A decode token drags its entire KV cache along; a prefill-chunk token does not. So the two kinds of token are not interchangeable:

256 decode tokens and a 512-token chunk have comparable token counts and sit on opposite sides of the ridge. Tokens-per-step is the first-order knob; the KV column is the second-order term that decides whether the knob still works. The practical reading is narrower than the tidy sentence and more useful: mixing a prefill chunk into a decode batch is what lifts a step over the ridge — the decode tokens alone never will.

What I would want to measure

• Whether the ceiling is observable. Everything above is an analytic model with two fudge factors (MFU, bandwidth utilisation). Does achieved decode throughput actually flatten near $g(6d/S+1)$ on real hardware, or does something else bind first?
• Where the mixed-batch optimum sits. The table says a chunk rescues a decode batch, but the chunk also adds TPOT. There is an optimum in (batch, chunk) and I have not seen it mapped for a fixed SLO pair.
• Whether the exponential-length assumption survives. Occupancy $1/H_B$ is exact for iid exponential outputs. Real output lengths are neither iid nor exponential — chat traffic is bimodal, and requests arrive correlated. I would expect static batching to look worse, not better, but that is a guess.

Written while working through the arithmetic rather than after; the derivations are mine and reproducible, the efficiency assumptions are guesses, and I would like to be corrected on the third bullet in particular.