{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/gpu","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/gpu/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/gpu/"},"term":{"en":"Graphics processing unit (GPU)","da":"Grafikprocessor (GPU)"},"aka":{"en":["GPU"],"da":["GPU"]},"domain":["ai"],"cluster":"ai-infrastructure","layer":"hardware","status":"current","era":1999,"summary":{"en":"A chip first built to draw screen images that does thousands of small sums at once, which is why it now runs most AI work.","da":"En chip bygget til skærmbilleder, der laver tusindvis af små beregninger på én gang og derfor nu driver det meste AI-arbejde."},"body":{"formal":{"en":"A processor made of thousands of simple cores that carry out the same kind of number work side by side, with its own fast memory; neural networks suit it because their work is mostly large grids of numbers being multiplied together.","da":"En processor bestående af tusindvis af simple kerner, der udfører den samme slags talarbejde side om side, med sin egen hurtige hukommelse; neurale netværk passer godt til den, fordi deres arbejde mest består af store tabeller af tal, der ganges sammen."},"plain":{"en":"Like a school hall of a thousand pupils each doing one easy sum at the same moment, instead of one professor working through the whole pile alone.","da":"Som en skolesal med tusind elever, der hver regner ét let stykke på samme tid, i stedet for én professor, der arbejder sig gennem hele bunken alene."},"inPractice":{"en":"A region's IT operations lead plans to run an open model in-house for hospital staff and finds the real limit is the memory on each GPU - the model weights must fit there, or answers slow to a crawl.","da":"En IT-driftsansvarlig i en region vil køre en åben model internt for hospitalets personale og opdager, at den reelle grænse er hukommelsen på hver GPU - modelvægtene skal kunne ligge der, ellers bliver svarene meget langsomme."},"whyItMatters":{"en":"GPUs are scarce and costly, so who can get them shapes who can build and run large models, what AI services cost, and how much power they use.","da":"GPU'er er knappe og dyre, så hvem der kan skaffe dem, afgør, hvem der kan bygge og køre store modeller, hvad AI-tjenester koster, og hvor meget strøm de bruger."}},"deepDive":{"en":"A modern GPU is a throughput machine built from many streaming multiprocessors (SMs, in NVIDIA's terminology; AMD calls them compute units). Each SM schedules threads in groups of 32 called warps and executes them in a SIMT (single instruction, multiple threads) fashion: all threads of a warp step through the same instruction on different data, and divergent branches are serialised. Instead of hiding memory latency with large caches and out-of-order execution as a CPU does, the GPU keeps thousands of warps resident and switches between them every cycle, so that while some wait for memory others compute. The price is that code must expose massive, regular data parallelism to run well.\n\nThe shift from graphics to general computing came in stages: programmable shaders in the early 2000s, NVIDIA's CUDA platform in 2006-2007, and the 2012 AlexNet result, trained on two consumer GPUs, which showed that deep neural networks were practical on this hardware. Since the Volta generation (2017), NVIDIA GPUs contain dedicated tensor cores that perform small matrix multiply-accumulate operations per instruction in reduced precision (FP16, BF16, and on later generations FP8 and FP4), delivering most of the chip's advertised FLOPS. Software rarely targets the hardware directly; frameworks such as PyTorch call vendor libraries (cuBLAS, cuDNN, or ROCm on AMD) and increasingly compiler-generated kernels.\n\nPerformance is best reasoned about with the roofline model: a kernel is compute-bound if its arithmetic intensity (FLOPs per byte moved from memory) exceeds the ratio of peak FLOPS to memory bandwidth, and memory-bound otherwise. Large-batch training and the prefill phase of LLM inference are mostly compute-bound; token-by-token decoding at small batch sizes is memory-bound, because every generated token requires reading all weights and the KV cache from high-bandwidth memory (HBM). This is why HBM capacity and bandwidth, not peak FLOPS, often decide which model fits on which card and how fast it answers, and why quantization and batching matter so much.\n\nModels larger than one device are split across GPUs with tensor, pipeline, data or expert parallelism, which makes the interconnect critical: NVLink and NVSwitch within a server, and InfiniBand or RDMA-capable Ethernet between servers. Common misconceptions are that more GPUs always means proportionally more speed (communication overhead and memory limits break linear scaling) and that consumer and data-centre cards are interchangeable (they differ in memory size, interconnect, ECC and licensing terms for data-centre use). Compared with a TPU, the GPU is more general and is sold by several vendors with a broad software ecosystem, whereas the TPU is a Google-designed accelerator centred on systolic matrix units and mainly offered as a cloud service. Operationally, power and cooling are real constraints: current data-centre GPUs draw several hundred watts to over a kilowatt each, and dense racks need liquid cooling.","da":"En moderne GPU er en maskine bygget til gennemløb og består af mange streaming multiprocessors (SM'er i NVIDIAs terminologi; AMD kalder dem compute units). Hver SM planlægger tråde i grupper på 32, kaldet warps, og afvikler dem efter SIMT-princippet (single instruction, multiple threads): alle tråde i en warp udfører den samme instruktion på forskellige data, og forgreninger, der divergerer, afvikles efter hinanden. I stedet for at skjule hukommelseslatens med store caches og out-of-order-afvikling, som en CPU gør, holder GPU'en tusindvis af warps klar og skifter mellem dem hver clockcyklus, så nogle regner, mens andre venter på hukommelsen. Prisen er, at koden skal rumme massiv, regelmæssig dataparallelisme for at køre godt.\n\nSkiftet fra grafik til generel beregning skete gradvist: programmerbare shadere i begyndelsen af 2000'erne, NVIDIAs CUDA-platform i 2006-2007 og AlexNet-resultatet i 2012, trænet på to almindelige forbruger-GPU'er, som viste, at dybe neurale netværk var praktisk mulige på denne hardware. Siden Volta-generationen (2017) har NVIDIAs GPU'er haft dedikerede tensor cores, der udfører små matrix-multiply-accumulate-operationer pr. instruktion med reduceret præcision (FP16, BF16 og på nyere generationer FP8 og FP4), og som står for størstedelen af chippens oplyste FLOPS. Software programmerer sjældent hardwaren direkte; frameworks som PyTorch kalder leverandørbiblioteker (cuBLAS, cuDNN eller ROCm hos AMD) og i stigende grad kernels genereret af compilere.\n\nYdelse analyseres bedst med roofline-modellen: en kernel er compute-bound, hvis dens aritmetiske intensitet (FLOPs pr. byte flyttet fra hukommelsen) overstiger forholdet mellem maksimal FLOPS og hukommelsesbåndbredde, og ellers memory-bound. Træning med store batches og prefill-fasen i LLM-inferens er overvejende compute-bound; generering token for token med små batches er memory-bound, fordi hvert nyt token kræver, at alle vægte og hele KV-cachen læses fra high-bandwidth memory (HBM). Derfor er det ofte HBM-kapacitet og -båndbredde og ikke maksimal FLOPS, der afgør, hvilken model der kan ligge på hvilket kort, og hvor hurtigt den svarer, og derfor betyder kvantisering og batching så meget.\n\nModeller, der er større end én enhed, fordeles over flere GPU'er med tensor-, pipeline-, data- eller ekspertparallelisme, hvilket gør forbindelsen mellem chippene afgørende: NVLink og NVSwitch inden for en server og InfiniBand eller RDMA-kompatibelt Ethernet mellem servere. Udbredte misforståelser er, at flere GPU'er altid giver tilsvarende mere fart (kommunikationsomkostninger og hukommelsesgrænser bryder den lineære skalering), og at forbruger- og datacenterkort kan bruges i flæng (de adskiller sig på hukommelsesstørrelse, interconnect, ECC og licensvilkår for brug i datacentre). Sammenlignet med en TPU er GPU'en mere generel og sælges af flere leverandører med et bredt softwareøkosystem, mens TPU'en er en Google-designet accelerator bygget op om systoliske matrixenheder og primært udbydes som cloudtjeneste. I driften er strøm og køling reelle begrænsninger: aktuelle datacenter-GPU'er trækker fra flere hundrede watt til over en kilowatt hver, og tætte racks kræver væskekøling."},"edges":[{"type":"alternative-to","to":"ai/tpu","why":{"en":"Both are chips for the heavy number work of neural networks; the GPU is general and sold by several makers, the TPU is Google's own design, mostly rented through its cloud.","da":"Begge er chips til det tunge talarbejde i neurale netværk; GPU'en er generel og sælges af flere producenter, TPU'en er Googles eget design, som mest lejes gennem dets cloud."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/model-training","why":{"en":"Training a modern model means huge amounts of repeated number work, which is only practical when spread across many GPUs.","da":"At træne en moderne model kræver enorme mængder gentaget talarbejde, hvilket kun er praktisk, når det fordeles over mange GPU'er."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/inference","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/model-serving","confidence":"high","strength":"normal"}],"depth":0,"sources":[{"title":"NVIDIA (1999), GeForce 256 announcement - first chip marketed as a \"GPU\"","tier":"official-doc","publisher":"NVIDIA"},{"title":"Goodfellow, Bengio & Courville, Deep Learning (ch. 12.1, large-scale deep learning)","tier":"textbook","publisher":"MIT Press"}],"draft":true}