{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/inference","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/inference/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/inference/"},"term":{"en":"Inference","da":"Inferens"},"aka":{"en":[],"da":[]},"domain":["ai"],"cluster":"ml-fundamentals","layer":"inference","status":"current","summary":{"en":"Using an already trained model to produce an answer for new input, which is what happens each time you ask a chat assistant something.","da":"At bruge en færdigtrænet model til at give et svar på nyt input - det, der sker, hver gang du spørger en chatassistent om noget."},"body":{"formal":{"en":"The stage where a trained model, with its weights fixed, is run on new input to produce an output such as a label, a score or generated text; the model does not learn from the input by doing so.","da":"Det trin, hvor en trænet model med faste vægte køres på nyt input for at give et resultat som en kategori, en score eller genereret tekst; modellen lærer ikke noget af inputtet undervejs."},"plain":{"en":"Like a trained doctor looking at a new patient; the years of study are over, and now the knowledge is simply being used.","da":"Som en uddannet læge, der ser på en ny patient - årene med studier er forbi, nu bliver viden bare brugt."},"inPractice":{"en":"When a clerk in a municipality pastes a contract into an online AI tool and asks for a summary, the text is sent to the provider's servers abroad, where inference runs.","da":"Når en medarbejder i en kommune indsætter en kontrakt i et online AI-værktøj og beder det opsummere den, sendes teksten til udbyderens servere i udlandet, hvor inferensen kører."},"whyItMatters":{"en":"Where inference runs decides where your data travels and who can see it; a local model and a foreign cloud service carry very different risks.","da":"Hvor inferensen kører, afgør, hvor dine data rejser hen, og hvem der kan se dem - en lokal model og en udenlandsk cloudtjeneste har vidt forskellige risici."}},"deepDive":{"en":"Mechanically, inference is a forward pass: the input is encoded (tokenised, normalised, resized), multiplied through the frozen weights layer by layer, and turned into an output by a final head such as a softmax over classes or over a vocabulary. No gradients are computed and no optimiser state is kept, so memory per request is dominated by the weights themselves plus activations. That is why inference can run on hardware far smaller than the cluster used for model training, and why techniques such as quantization (storing weights in INT8 or 4-bit formats) and knowledge distillation target inference specifically.\n\nAutoregressive language models make inference a loop rather than a single pass. In the prefill phase the whole prompt is processed in parallel and the attention keys and values for every token are stored in a KV cache; in the decode phase the model produces one token at a time, each step reading the entire cache. Prefill is compute-bound and determines time-to-first-token; decode is memory-bandwidth-bound and determines tokens per second. Serving systems therefore rely on continuous batching (adding and removing requests from a running batch between decode steps), paged KV-cache memory and speculative decoding, where a small draft model proposes several tokens that the large model verifies in one pass.\n\nOutput is not always deterministic. Sampling settings such as temperature and top-p deliberately introduce randomness, and even at temperature 0 floating-point non-associativity across different batch sizes and GPU kernels can change results. Since around 2024, reasoning models have also shifted cost towards inference: they spend extra tokens on intermediate reasoning before answering, so the cost of a single request can vary by orders of magnitude.\n\nA common confusion is with statistical inference, which means drawing conclusions about a population from a sample (confidence intervals, hypothesis tests). In machine learning the word simply means running the model. Another misconception is that a deployed model learns from user input during inference; it does not, unless a provider separately collects the prompts and uses them in a later training run, which is a contractual and data protection question rather than a technical property of inference.\n\nFrom a security and compliance angle, inference is where the data actually flows. Every prompt and document sent to a hosted model leaves the organisation, so the location of the endpoint determines whether GDPR Chapter V rules on third-country transfers apply. Inference endpoints are also the attack surface for prompt injection, model extraction via repeated queries, and membership inference attacks that test whether a given record was in the training data.","da":"Mekanisk er inferens et forward pass: inputtet kodes (tokeniseres, normaliseres, skaleres), ganges gennem de frosne vægte lag for lag og omsættes til et resultat af et afsluttende lag, fx en softmax over klasser eller over et ordforråd. Der beregnes ingen gradienter, og der gemmes ingen optimeringstilstand, så hukommelsesforbruget pr. forespørgsel domineres af selve vægtene plus aktiveringerne. Derfor kan inferens køre på langt mindre hardware end den klynge, der blev brugt til modeltræning, og derfor er teknikker som kvantisering (vægte gemt i INT8 eller 4-bit-formater) og knowledge distillation rettet netop mod inferens.\n\nAutoregressive sprogmodeller gør inferens til en løkke frem for ét gennemløb. I prefill-fasen behandles hele prompten parallelt, og attention-nøgler og -værdier for hvert token gemmes i en KV-cache; i decode-fasen danner modellen ét token ad gangen, og hvert skridt læser hele cachen. Prefill er begrænset af regnekraft og afgør time-to-first-token; decode er begrænset af hukommelsesbåndbredde og afgør antal tokens pr. sekund. Serving-systemer bruger derfor continuous batching (forespørgsler lægges til og fjernes fra en kørende batch mellem decode-skridt), sideopdelt KV-cache-hukommelse og speculative decoding, hvor en lille udkastmodel foreslår flere tokens, som den store model verificerer i ét gennemløb.\n\nResultatet er ikke altid deterministisk. Samplingindstillinger som temperatur og top-p tilføjer bevidst tilfældighed, og selv ved temperatur 0 kan flydende-komma-aritmetikkens manglende associativitet på tværs af batchstørrelser og GPU-kerner ændre resultatet. Siden omkring 2024 har ræsonnerende modeller desuden flyttet omkostninger over på inferens: de bruger ekstra tokens på mellemregninger, før de svarer, så prisen for én forespørgsel kan variere med flere størrelsesordener.\n\nEn udbredt forveksling er med statistisk inferens, som betyder at drage konklusioner om en population ud fra en stikprøve (konfidensintervaller, hypotesetest). I maskinlæring betyder ordet blot at køre modellen. En anden misforståelse er, at en udrullet model lærer af brugerens input under inferens; det gør den ikke, medmindre udbyderen særskilt indsamler prompterne og bruger dem i en senere træningskørsel, og det er et kontraktligt og databeskyttelsesretligt spørgsmål, ikke en teknisk egenskab ved inferens.\n\nSet fra sikkerhed og compliance er inferens der, hvor data faktisk flyder. Hver prompt og hvert dokument, der sendes til en hostet model, forlader organisationen, så endpointets placering afgør, om databeskyttelsesforordningens kapitel V om overførsel til tredjelande finder anvendelse. Inferens-endpoints er også angrebsfladen for prompt injection, modeludtrækning via gentagne forespørgsler og membership inference-angreb, der tester, om en bestemt post indgik i træningsdata."},"edges":[{"type":"part-of","to":"ai/machine-learning","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/model-training","why":{"en":"Training is the slow, one-off stage where the model learns; inference is every later use, where it only applies what it learned.","da":"Træning er den langsomme engangsfase, hvor modellen lærer; inferens er al senere brug, hvor den kun anvender det lærte."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/mixture-of-experts","why":{"en":"Only a few experts run for each token, so inference costs far less than the model's total size suggests.","da":"Kun få eksperter kører for hvert token, så inferens koster langt mindre, end modellens samlede størrelse antyder."},"confidence":"medium","strength":"normal"}],"depth":0,"sources":[{"title":"ISO/IEC 22989:2022, Artificial intelligence concepts and terminology","url":"https://www.iso.org/standard/74296.html","tier":"standard","publisher":"ISO/IEC"},{"title":"Goodfellow, Bengio & Courville (2016), Deep Learning","url":"https://www.deeplearningbook.org/","tier":"textbook","publisher":"MIT Press"}],"draft":true}