Throughput

Throughput must always be stated with its unit and its conditions. For LLM services the common units are output tokens per second, total tokens per second (input plus output, which flatters prompt-heavy workloads because prefill processes many tokens in parallel), and requests per second; each can be reported per GPU, per server or per cluster. The figure depends heavily on the workload shape: prompt and output lengths, concurrency, the model and precision, and whether prefix caching hits. Two numbers measured with different input/output ratios are not comparable, which is why benchmark suites fix these parameters. MLPerf Inference, for instance, distinguishes an Offline scenario, which measures raw throughput with all queries available at once, from a Server scenario, which measures the highest query rate sustainable while meeting latency constraints.

The physics of LLM throughput follows from the decode phase being memory-bandwidth-bound. Generating one token for one sequence requires streaming all model weights from HBM; generating one token for 64 sequences in the same step requires reading the weights once and reusing them 64 times. Batching therefore raises aggregate tokens per second nearly linearly until the step becomes compute-bound or KV-cache memory runs out. This is why continuous batching, paged KV-cache management, grouped-query attention and quantization, all of which free memory for more concurrent sequences, are primarily throughput techniques.

Throughput and latency are coupled, not independent. By Little's law, concurrency = throughput × time in system, so for a fixed number of concurrent users, higher throughput means lower latency; but pushing for maximum throughput by growing batches makes each step slower and lengthens queues, raising per-request latency. The usual way to report the trade-off is a curve of throughput against a latency percentile at increasing load, and the operationally relevant point is the maximum throughput at which the latency SLO still holds. The DistServe paper (Zhong et al., 2024) framed serving performance as "goodput": the maximum request rate at which a target share of requests complete within their TTFT and TPOT targets, as opposed to raw tokens per second, which can look excellent while users time out.

Common mistakes include quoting peak vendor numbers measured at unrealistically high batch sizes, confusing throughput with bandwidth (the capacity of a link rather than the useful work completed), measuring with a load generator that is itself the bottleneck, and ignoring the input side, where long prompts consume prefill capacity that never appears in output-token counts. For capacity planning, throughput at the SLO translates directly into cost per million tokens and the number of accelerators needed for peak load.