Model serving

A model-serving stack has recognisable layers. At the bottom is an inference engine that owns the GPU: it loads weights, manages the KV cache, schedules requests and runs optimised kernels. Common LLM engines include vLLM, SGLang, NVIDIA TensorRT-LLM and llama.cpp; general-purpose servers such as NVIDIA Triton Inference Server host many model types behind one process. Above that sits an API layer, very often an OpenAI-compatible HTTP interface with server-sent events for streaming, then a gateway or router that handles authentication, quotas, model routing and logging, and an orchestration layer (for example Kubernetes with KServe) that places replicas on GPU nodes and scales them.

The decisive technique for LLMs is continuous batching, also called iteration-level scheduling, introduced by the Orca system (Yu et al., OSDI 2022). Classic static batching waits for a batch to fill and holds all slots until the longest sequence finishes; continuous batching admits and retires sequences at every decode step, keeping the GPU busy despite widely varying output lengths. Combined with paged KV-cache management, prefix caching, chunked prefill (splitting long prompts so they do not stall ongoing decodes) and speculative decoding, it typically raises throughput several-fold over naive serving. Large deployments increasingly disaggregate prefill and decode onto separate GPU pools, since the first is compute-bound and the second memory-bound.

Models that exceed one GPU's memory are sharded. Tensor parallelism splits each layer's matrices across GPUs in a node and needs a fast interconnect; pipeline parallelism places consecutive layers on different devices; expert parallelism distributes the experts of mixture-of-experts models. Capacity is set by memory: weights plus KV cache for the target concurrency and context length must fit, with headroom for activations. Autoscaling is harder than for stateless web services because cold starts involve pulling tens or hundreds of gigabytes of weights and warming kernels, so operators keep warm pools, scale on queue depth or KV-cache utilisation rather than CPU, and set explicit maximum context and output lengths.

Operational concerns go beyond speed. Serving is where the service-level objectives are enforced (TTFT and TPOT percentiles, error rate), where model versions are rolled out with canaries and rolled back, and where observability captures tokens, latency and cost per tenant. It is also a security boundary: prompts and outputs are often personal or confidential data, so logging, retention, data residency and tenant isolation of caches must be designed deliberately, and the endpoint needs rate limiting and authentication like any other API. Serving differs from training in that it is a continuously available, latency-sensitive production service rather than a batch job, and for a widely used model it can account for more of the lifetime cost than training did.