Latency

Latency is a distribution, not a number. Serious measurements report percentiles such as p50, p95, p99 and p99.9 over a defined window, because averages hide the tail and the tail is what users of fan-out systems experience: if one page request calls 100 backends in parallel, the page is as slow as the slowest, so a 1-in-100 slow backend affects most page loads (Dean and Barroso, "The Tail at Scale", 2013). Percentiles cannot be averaged across hosts or time windows; they must be computed from merged histograms (for example HDR histograms or t-digests). Load generators that wait for each response before sending the next under-report tail latency during stalls, a flaw known as coordinated omission.

Latency is tied to load through queueing theory. Little's law, L = λW, links the average number of requests in the system to arrival rate and time in the system. In simple queueing models waiting time grows non-linearly as utilisation approaches 100% (for an M/M/1 queue the mean time in system is 1/(μ − λ)), which is why a service that is fine at 60% utilisation can become unusable at 90%. Capacity planning therefore sets latency targets at a percentile and derives the utilisation ceiling from them, rather than the other way round.

For LLM services, end-to-end latency decomposes into network and queueing time, time to first token (dominated by prefill of the prompt), and the decode phase, often expressed as time per output token (TPOT) or inter-token latency (ITL). A useful approximation is E2E ≈ TTFT + TPOT × (output tokens − 1). The levers differ per component: prompt length and prompt caching affect TTFT; model size, quantization, speculative decoding and batch size affect TPOT; and output length is often the largest single factor, which is why capping or shortening answers reduces latency more than hardware changes do. Reasoning models add hidden thinking tokens, so perceived latency can be much higher than the visible answer length suggests.

The central trade-off is with throughput. Larger batches raise tokens per second per GPU but make each sequence step slower and add queueing, so serving systems tune batch limits against a latency SLO. Agentic workflows amplify the problem: an agent that makes ten sequential model and tool calls multiplies per-call latency, so their design favours parallel calls, smaller models for routing steps and streaming. When reporting latency, state where it is measured (client, gateway or model server), which percentile, under what concurrency, and with which prompt and output lengths; numbers without that context are not comparable.