{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/latency","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/latency/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/latency/"},"term":{"en":"Latency","da":"Latens"},"aka":{"en":[],"da":[]},"domain":["ai","cs"],"cluster":"ai-infrastructure","layer":"inference","status":"current","summary":{"en":"How long one request has to wait from being sent until its answer arrives - for an AI chat, the pause before and while it replies.","da":"Hvor længe én forespørgsel skal vente, fra den sendes, til svaret kommer - for en AI-chat pausen før og mens den svarer."},"body":{"formal":{"en":"The time from a request leaving the sender until the matching response arrives, usually reported as a typical value and a slow-end value, such as the time 99 in 100 requests beat; for a language model it covers queueing, reading the prompt and producing each token.","da":"Tiden fra en forespørgsel forlader afsenderen, til det tilhørende svar er modtaget, typisk angivet som en normal værdi og en værdi for de langsomme, fx den tid, 99 ud af 100 forespørgsler er hurtigere end; for en sprogmodel dækker den kø, læsning af prompten og frembringelse af hvert token."},"plain":{"en":"Like the wait between ordering a coffee and holding it in your hand - it says nothing about how many coffees the café makes in an hour.","da":"Som ventetiden fra man bestiller en kaffe, til man står med den i hånden - den siger intet om, hvor mange kaffer caféen laver i timen."},"inPractice":{"en":"A housing association's web manager moves its tenant chat to a smaller model because tenants gave up when answers took eight seconds to start; the smaller model begins replying in under one.","da":"Den webansvarlige i en boligforening flytter lejerchatten over på en mindre model, fordi lejerne gav op, når svarene var otte sekunder om at begynde; den mindre model begynder at svare på under ét."},"whyItMatters":{"en":"People judge a service by how quickly it responds, and tools that call a model many times in a row feel every extra delay add up.","da":"Folk bedømmer en tjeneste på, hvor hurtigt den svarer, og værktøjer, der kalder en model mange gange i træk, mærker hver ekstra forsinkelse lægge sig oven i hinanden."}},"deepDive":{"en":"Latency is a distribution, not a number. Serious measurements report percentiles such as p50, p95, p99 and p99.9 over a defined window, because averages hide the tail and the tail is what users of fan-out systems experience: if one page request calls 100 backends in parallel, the page is as slow as the slowest, so a 1-in-100 slow backend affects most page loads (Dean and Barroso, \"The Tail at Scale\", 2013). Percentiles cannot be averaged across hosts or time windows; they must be computed from merged histograms (for example HDR histograms or t-digests). Load generators that wait for each response before sending the next under-report tail latency during stalls, a flaw known as coordinated omission.\n\nLatency is tied to load through queueing theory. Little's law, L = λW, links the average number of requests in the system to arrival rate and time in the system. In simple queueing models waiting time grows non-linearly as utilisation approaches 100% (for an M/M/1 queue the mean time in system is 1/(μ − λ)), which is why a service that is fine at 60% utilisation can become unusable at 90%. Capacity planning therefore sets latency targets at a percentile and derives the utilisation ceiling from them, rather than the other way round.\n\nFor LLM services, end-to-end latency decomposes into network and queueing time, time to first token (dominated by prefill of the prompt), and the decode phase, often expressed as time per output token (TPOT) or inter-token latency (ITL). A useful approximation is E2E ≈ TTFT + TPOT × (output tokens − 1). The levers differ per component: prompt length and prompt caching affect TTFT; model size, quantization, speculative decoding and batch size affect TPOT; and output length is often the largest single factor, which is why capping or shortening answers reduces latency more than hardware changes do. Reasoning models add hidden thinking tokens, so perceived latency can be much higher than the visible answer length suggests.\n\nThe central trade-off is with throughput. Larger batches raise tokens per second per GPU but make each sequence step slower and add queueing, so serving systems tune batch limits against a latency SLO. Agentic workflows amplify the problem: an agent that makes ten sequential model and tool calls multiplies per-call latency, so their design favours parallel calls, smaller models for routing steps and streaming. When reporting latency, state where it is measured (client, gateway or model server), which percentile, under what concurrency, and with which prompt and output lengths; numbers without that context are not comparable.","da":"Latens er en fordeling, ikke et enkelt tal. Seriøse målinger angiver percentiler som p50, p95, p99 og p99,9 over et defineret tidsvindue, fordi gennemsnit skjuler halen, og det er halen, brugere af fan-out-systemer mærker: hvis én sidevisning kalder 100 backends parallelt, er siden lige så langsom som den langsomste, så en backend, der er langsom én gang ud af 100, rammer de fleste sidevisninger (Dean og Barroso, \"The Tail at Scale\", 2013). Percentiler kan ikke midles på tværs af servere eller tidsvinduer; de skal beregnes ud fra sammenlagte histogrammer (fx HDR-histogrammer eller t-digests). Belastningsværktøjer, der venter på hvert svar, før de sender det næste, underrapporterer halelatensen under stop, en fejl kendt som coordinated omission.\n\nLatens hænger sammen med belastning gennem køteori. Littles lov, L = λW, forbinder det gennemsnitlige antal forespørgsler i systemet med ankomstraten og tiden i systemet. I simple kømodeller vokser ventetiden ikke-lineært, når udnyttelsen nærmer sig 100 % (for en M/M/1-kø er den gennemsnitlige tid i systemet 1/(μ − λ)), og derfor kan en tjeneste, der kører fint ved 60 % udnyttelse, blive ubrugelig ved 90 %. Kapacitetsplanlægning fastsætter derfor latensmål ved en bestemt percentil og udleder loftet for udnyttelse derfra, ikke omvendt.\n\nFor LLM-tjenester kan end-to-end-latensen opdeles i netværks- og køtid, tid til første token (domineret af prefill af prompten) og decode-fasen, ofte udtrykt som tid pr. output-token (TPOT) eller inter-token-latens (ITL). En brugbar tilnærmelse er E2E ≈ TTFT + TPOT × (antal output-tokens − 1). Håndtagene er forskellige for hver del: promptlængde og prompt caching påvirker TTFT; modelstørrelse, kvantisering, speculative decoding og batchstørrelse påvirker TPOT; og svarets længde er ofte den største enkeltfaktor, hvilket er grunden til, at et loft over eller kortere svar sænker latensen mere end ny hardware. Ræsonnerende modeller tilføjer skjulte tænke-tokens, så den oplevede latens kan være langt højere, end længden af det synlige svar antyder.\n\nDet centrale kompromis er med gennemløb. Større batches hæver antallet af tokens pr. sekund pr. GPU, men gør hvert trin for den enkelte sekvens langsommere og giver mere kø, så serving-systemer justerer batchgrænserne mod et latens-SLO. Agentbaserede arbejdsgange forstærker problemet: en agent, der foretager ti model- og værktøjskald efter hinanden, ganger latensen pr. kald op, så designet foretrækker parallelle kald, mindre modeller til routing-trin og streaming. Når latens rapporteres, skal det fremgå, hvor den er målt (klient, gateway eller modelserver), hvilken percentil, ved hvilken samtidighed og med hvilke prompt- og svarlængder; tal uden den kontekst kan ikke sammenlignes."},"edges":[{"type":"contrasts-with","to":"ai/throughput","why":{"en":"Latency is how long one request waits; throughput is how much total work gets done per second. Grouping requests together often raises throughput while making each one wait longer.","da":"Latens er, hvor længe én forespørgsel venter; gennemløb er, hvor meget arbejde der i alt klares pr. sekund. At samle forespørgsler i grupper hæver ofte gennemløbet, men får hver enkelt til at vente længere."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"platform/service-level-objective","why":{"en":"Speed goals for a service are usually written as a latency limit, such as 95 in 100 answers starting within one second.","da":"Hastighedsmål for en tjeneste skrives normalt som en latensgrænse, fx at 95 ud af 100 svar begynder inden for ét sekund."},"confidence":"high","strength":"normal"},{"type":"used-with","to":"platform/monitoring","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/model-serving","confidence":"high","strength":"normal"}],"depth":0,"sources":[{"title":"Beyer et al. (eds.), Site Reliability Engineering (ch. 4, Service Level Objectives)","tier":"textbook","publisher":"O'Reilly / Google"},{"title":"Hennessy & Patterson, Computer Architecture: A Quantitative Approach (ch. 1)","tier":"textbook","publisher":"Morgan Kaufmann"}],"draft":true}