Inference

Mechanically, inference is a forward pass: the input is encoded (tokenised, normalised, resized), multiplied through the frozen weights layer by layer, and turned into an output by a final head such as a softmax over classes or over a vocabulary. No gradients are computed and no optimiser state is kept, so memory per request is dominated by the weights themselves plus activations. That is why inference can run on hardware far smaller than the cluster used for model training, and why techniques such as quantization (storing weights in INT8 or 4-bit formats) and knowledge distillation target inference specifically.

Autoregressive language models make inference a loop rather than a single pass. In the prefill phase the whole prompt is processed in parallel and the attention keys and values for every token are stored in a KV cache; in the decode phase the model produces one token at a time, each step reading the entire cache. Prefill is compute-bound and determines time-to-first-token; decode is memory-bandwidth-bound and determines tokens per second. Serving systems therefore rely on continuous batching (adding and removing requests from a running batch between decode steps), paged KV-cache memory and speculative decoding, where a small draft model proposes several tokens that the large model verifies in one pass.

Output is not always deterministic. Sampling settings such as temperature and top-p deliberately introduce randomness, and even at temperature 0 floating-point non-associativity across different batch sizes and GPU kernels can change results. Since around 2024, reasoning models have also shifted cost towards inference: they spend extra tokens on intermediate reasoning before answering, so the cost of a single request can vary by orders of magnitude.

A common confusion is with statistical inference, which means drawing conclusions about a population from a sample (confidence intervals, hypothesis tests). In machine learning the word simply means running the model. Another misconception is that a deployed model learns from user input during inference; it does not, unless a provider separately collects the prompts and uses them in a later training run, which is a contractual and data protection question rather than a technical property of inference.

From a security and compliance angle, inference is where the data actually flows. Every prompt and document sent to a hosted model leaves the organisation, so the location of the endpoint determines whether GDPR Chapter V rules on third-country transfers apply. Inference endpoints are also the attack surface for prompt injection, model extraction via repeated queries, and membership inference attacks that test whether a given record was in the training data.