{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/model-serving","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/model-serving/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/model-serving/"},"term":{"en":"Model serving","da":"Modelservering (model serving)"},"aka":{"en":["inference serving","model hosting"],"da":["inference serving","modelhosting"]},"domain":["ai"],"cluster":"ai-infrastructure","layer":"inference","status":"current","summary":{"en":"Keeping a trained model running on machines so other programs can send it questions over an API and get answers back.","da":"At holde en trænet model kørende på maskiner, så andre programmer kan sende den spørgsmål via et API og få svar tilbage."},"body":{"formal":{"en":"The work of loading model weights onto suitable hardware and exposing the model through an API that accepts requests, runs inference, and returns results, while handling many users at once, spreading load, and keeping the service up.","da":"Arbejdet med at indlæse modelvægte på egnet hardware og gøre modellen tilgængelig gennem et API, der tager imod forespørgsler, kører inferens og returnerer resultater, mens det håndterer mange brugere på én gang, fordeler belastningen og holder tjenesten oppe."},"plain":{"en":"Like a power station - the design work is done; the daily job is keeping power flowing to every home, however many people switch on at the same moment.","da":"Som et kraftværk - konstruktionen er færdig; det daglige arbejde er at sikre strøm til hvert hjem, uanset hvor mange der tænder på samme tid."},"inPractice":{"en":"A Danish university's IT operations lead serves an open model to 30,000 students and staff through one internal API; in exam weeks the queue grows, so she adds two GPUs and caps the length of each answer.","da":"IT-driftsansvarlig på et dansk universitet stiller en åben model til rådighed for 30.000 studerende og ansatte gennem ét internt API; i eksamensugerne vokser køen, så hun sætter to GPU'er mere ind og begrænser længden af hvert svar."},"whyItMatters":{"en":"Most of what an AI service costs to run, how fast it feels, and where its data goes is decided here, not during training.","da":"Det meste af, hvad en AI-tjeneste koster at køre, hvor hurtig den føles, og hvor dens data havner, afgøres her - ikke under træningen."}},"deepDive":{"en":"A model-serving stack has recognisable layers. At the bottom is an inference engine that owns the GPU: it loads weights, manages the KV cache, schedules requests and runs optimised kernels. Common LLM engines include vLLM, SGLang, NVIDIA TensorRT-LLM and llama.cpp; general-purpose servers such as NVIDIA Triton Inference Server host many model types behind one process. Above that sits an API layer, very often an OpenAI-compatible HTTP interface with server-sent events for streaming, then a gateway or router that handles authentication, quotas, model routing and logging, and an orchestration layer (for example Kubernetes with KServe) that places replicas on GPU nodes and scales them.\n\nThe decisive technique for LLMs is continuous batching, also called iteration-level scheduling, introduced by the Orca system (Yu et al., OSDI 2022). Classic static batching waits for a batch to fill and holds all slots until the longest sequence finishes; continuous batching admits and retires sequences at every decode step, keeping the GPU busy despite widely varying output lengths. Combined with paged KV-cache management, prefix caching, chunked prefill (splitting long prompts so they do not stall ongoing decodes) and speculative decoding, it typically raises throughput several-fold over naive serving. Large deployments increasingly disaggregate prefill and decode onto separate GPU pools, since the first is compute-bound and the second memory-bound.\n\nModels that exceed one GPU's memory are sharded. Tensor parallelism splits each layer's matrices across GPUs in a node and needs a fast interconnect; pipeline parallelism places consecutive layers on different devices; expert parallelism distributes the experts of mixture-of-experts models. Capacity is set by memory: weights plus KV cache for the target concurrency and context length must fit, with headroom for activations. Autoscaling is harder than for stateless web services because cold starts involve pulling tens or hundreds of gigabytes of weights and warming kernels, so operators keep warm pools, scale on queue depth or KV-cache utilisation rather than CPU, and set explicit maximum context and output lengths.\n\nOperational concerns go beyond speed. Serving is where the service-level objectives are enforced (TTFT and TPOT percentiles, error rate), where model versions are rolled out with canaries and rolled back, and where observability captures tokens, latency and cost per tenant. It is also a security boundary: prompts and outputs are often personal or confidential data, so logging, retention, data residency and tenant isolation of caches must be designed deliberately, and the endpoint needs rate limiting and authentication like any other API. Serving differs from training in that it is a continuously available, latency-sensitive production service rather than a batch job, and for a widely used model it can account for more of the lifetime cost than training did.","da":"En stak til modelservering har genkendelige lag. Nederst ligger en inferensmotor, der ejer GPU'en: den indlæser vægte, håndterer KV-cachen, planlægger forespørgsler og kører optimerede kernels. Udbredte LLM-motorer er vLLM, SGLang, NVIDIA TensorRT-LLM og llama.cpp; generelle servere som NVIDIA Triton Inference Server kan huse mange modeltyper i én proces. Ovenover ligger et API-lag, meget ofte en OpenAI-kompatibel HTTP-grænseflade med server-sent events til streaming, derefter en gateway eller router, der står for autentificering, kvoter, routing mellem modeller og logning, og et orkestreringslag (fx Kubernetes med KServe), der placerer replikaer på GPU-noder og skalerer dem.\n\nDen afgørende teknik for LLM'er er continuous batching, også kaldet planlægning på iterationsniveau, som blev introduceret med Orca-systemet (Yu m.fl., OSDI 2022). Klassisk statisk batching venter på, at en batch bliver fyldt, og holder alle pladser, indtil den længste sekvens er færdig; continuous batching optager og afslutter sekvenser i hvert decode-trin og holder dermed GPU'en beskæftiget, selv om svarlængderne varierer meget. Sammen med sidebaseret styring af KV-cachen, prefix caching, chunked prefill (opdeling af lange prompter, så de ikke blokerer igangværende decode) og speculative decoding hæver det typisk gennemløbet flere gange i forhold til naiv servering. Store installationer adskiller i stigende grad prefill og decode på separate GPU-puljer, fordi den første er compute-bound og den anden memory-bound.\n\nModeller, der er større end én GPU's hukommelse, deles op. Tensorparallelisme fordeler hvert lags matricer over GPU'er i en node og kræver en hurtig interconnect; pipelineparallelisme placerer på hinanden følgende lag på forskellige enheder; ekspertparallelisme fordeler eksperterne i mixture-of-experts-modeller. Kapaciteten bestemmes af hukommelsen: vægte plus KV-cache til den ønskede samtidighed og kontekstlængde skal kunne ligge der med plads til aktiveringer. Autoskalering er sværere end for tilstandsløse webtjenester, fordi en kold start indebærer at hente ti- eller hundredvis af gigabyte vægte og varme kernels op, så driften holder varme puljer, skalerer på kødybde eller KV-cache-udnyttelse frem for CPU og sætter eksplicitte grænser for kontekst- og svarlængde.\n\nDriftshensynene rækker ud over hastighed. Det er i serveringslaget, at service level objectives håndhæves (percentiler for TTFT og TPOT, fejlrate), at modelversioner rulles ud med canaries og rulles tilbage, og at observability registrerer tokens, latens og omkostning pr. lejer. Det er også en sikkerhedsgrænse: prompter og svar er ofte personoplysninger eller fortrolige data, så logning, opbevaring, dataplacering og isolering af caches mellem lejere skal designes bevidst, og endpointet skal have rate limiting og autentificering som ethvert andet API. Modelservering adskiller sig fra træning ved at være en løbende tilgængelig, latensfølsom produktionstjeneste frem for et batchjob, og for en udbredt model kan den over levetiden koste mere, end træningen gjorde."},"edges":[{"type":"requires","to":"ai/inference","why":{"en":"Serving is inference made into a lasting service - the same step of running the model, repeated for many users around the clock.","da":"Modelservering er inferens gjort til en varig tjeneste - det samme trin, hvor modellen køres, gentaget for mange brugere døgnet rundt."},"confidence":"high","strength":"primary"},{"type":"requires","to":"cs/api","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/throughput","why":{"en":"A serving setup is judged on how many requests and tokens it can handle each second, since that decides how many machines, and how much money, it needs.","da":"En opsætning til modelservering bedømmes på, hvor mange forespørgsler og tokens den kan klare i sekundet, fordi det afgør, hvor mange maskiner - og hvor mange penge - den kræver."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"platform/monitoring","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/quantization","confidence":"high","strength":"normal"}],"depth":5,"sources":[{"title":"Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention","tier":"reference"},{"title":"NVIDIA Triton Inference Server documentation","tier":"official-doc","publisher":"NVIDIA"}],"draft":true}