Latency vs Throughput

Why they differ

Latency is how long one request waits; throughput is how much total work gets done per second. Grouping requests together often raises throughput while making each one wait longer.

Latency

AI hardware & serving

How long one request has to wait from being sent until its answer arrives - for an AI chat, the pause before and while it replies.

Formal

The time from a request leaving the sender until the matching response arrives, usually reported as a typical value and a slow-end value, such as the time 99 in 100 requests beat; for a language model it covers queueing, reading the prompt and producing each token.

In plain English

Like the wait between ordering a coffee and holding it in your hand - it says nothing about how many coffees the café makes in an hour.

In practice

A housing association's web manager moves its tenant chat to a smaller model because tenants gave up when answers took eight seconds to start; the smaller model begins replying in under one.

Why it matters

People judge a service by how quickly it responds, and tools that call a model many times in a row feel every extra delay add up.

Throughput

AI hardware & serving

How much work a system gets through per second in total - for AI services, often counted as tokens or requests handled each second.

Formal

The rate at which a system completes work, measured as units finished per unit of time, such as requests per second; for a language model service it is usually the total number of tokens produced per second across all users at once.

In plain English

Like counting how many cars cross a bridge each hour - a wider bridge lets more through, even if each car drives no faster.

In practice

An operations engineer at a Danish software house that hosts a chat tool for 30 municipalities measures that its GPU server produces 2,500 tokens a second in total - enough for about 100 users at once - and orders a second server before the Monday-morning peak.

Why it matters

It decides how many machines a service needs and what each answer costs, so it drives the price users pay and how a service copes with sudden crowds.

Shared connections

Model serving Time to first token (TTFT)Service level objective (SLO)