LLM inference in cards: the map, and scheduling

"Make inference faster" is not one problem. A serving stack loses time in six different places, and the technique that fixes one of them usually does nothing for the other five. This is a card series that goes through them one at a time — what the technique actually changes, and which number it moves.

Six entry points for optimising LLM inference: scheduling, caching, kernels and execution, quantization, decoding, parallelism and deployment. — The map. Six entry points, grouped by what they primarily change — they combine, and most production stacks use all six.

02 · Scheduling

Scheduling is the one that needs no new kernels and no change to the model: it is only a decision about which requests run in the same forward pass. It also has the clearest trade — batching more requests raises throughput, and the requests already mid-generation pay for it in the gap between their tokens.

Three steps, in the order the ideas arrived: batching reuses one read of the weights across several requests; continuous batching lets the batch change every iteration instead of every request; chunked prefill caps how much prefill any single iteration can contain, so a long prompt joining the batch cannot stall everyone else's decode. The three numbers they trade against each other are TTFT, ITL and throughput.

Series 02: Scheduling — how requests are batched and executed. — 01 · What scheduling covers

Prefill and decode explained. — 02 · Prefill and decode

GPU dataflow: HBM to on-chip storage to compute units; small-batch decode is memory-bandwidth-bound. — 03 · Why small-batch decode waits on memory

Batching reuses one read of the weights across three requests. — 04 · Batching reuses the weights

Static batching versus continuous batching across four iterations. — 05 · Static vs continuous batching

A long prefill joining a mixed batch widens the decoding request's inter-token latency. — 06 · What a long prefill costs

Chunked prefill: a 24-token prompt split across three iterations under a token budget of nine. — 07 · Chunked prefill and the token budget

Scheduling summary with TTFT, ITL and throughput. — 08 · The summary, and the three numbers

What comes next

The remaining five: caching (the KV cache is a capacity problem, and paging is what stops fragmentation from eating the capacity), kernels and execution, quantization, speculative decoding, and parallelism with prefill–decode disaggregation — which is the multi-machine version of the same fight chunked prefill settles inside one GPU.

Sources Orca (OSDI '22) — iteration-level scheduling · Sarathi / Sarathi-Serve (OSDI '24) — chunked prefill and mixed batching · vLLM optimization and tuning docs · NVIDIA GPU performance and matrix-multiplication guides · How To Scale Your Model.