Time to first token (TTFT)

TTFT is the sum of several stages: client-to-server network time, authentication and gateway overhead, tokenization, time spent queued until the scheduler admits the request, the prefill forward pass over the whole prompt, sampling of the first token, and the return trip of the first streamed chunk. Only one of those stages scales directly with model and prompt size. Prefill costs roughly 2 × parameters × prompt tokens in FLOPs for the dense layers, plus an attention term that grows quadratically with prompt length, and because prefill is compute-bound it benefits from raw accelerator FLOPS rather than memory bandwidth. A 100,000-token prompt can therefore take seconds to prefill even on fast hardware, whereas a short chat message is dominated by queueing and network time.

TTFT should be kept distinct from the metrics that describe the rest of the response. Time per output token (TPOT) is the average interval between subsequent tokens; inter-token latency (ITL) is the distribution of those intervals, where spikes reveal scheduling hiccups; end-to-end latency covers the complete response. A system can have excellent TTFT and poor TPOT or the reverse, and they respond to different optimisations. Where TTFT is measured also matters: client-side measurements include network and TLS setup, server-side measurements usually start when the request is received, and benchmark tools differ in whether an initial empty or role-only chunk counts as the first token.

The main levers on TTFT are prompt length, prefix or prompt caching (skipping prefill for a cached shared prefix), scheduling policy and capacity headroom. Serving engines face a tension: a new request's prefill competes with the decode steps of requests already running. Prioritising prefill improves TTFT but causes stalls for streaming users; chunked prefill, as in Sarathi-Serve, splits long prompts into pieces interleaved with decode steps to bound that interference, and disaggregated serving moves prefill to separate GPUs entirely. Tensor parallelism across more GPUs also shortens prefill for large models.

Two newer patterns complicate the metric. Reasoning models may spend many tokens thinking before the visible answer begins; if thinking is hidden or summarised, the user experiences time to first visible token, which can be far larger than the API-level TTFT. Tool-using agents and retrieval pipelines add retrieval and tool latency before the model call even starts. For voice interfaces, where a gap of around a second already feels unnatural, teams budget TTFT together with speech recognition and speech synthesis latency and often use smaller models or speculative early responses to meet the budget.