Graphics processing unit (GPU)

A modern GPU is a throughput machine built from many streaming multiprocessors (SMs, in NVIDIA's terminology; AMD calls them compute units). Each SM schedules threads in groups of 32 called warps and executes them in a SIMT (single instruction, multiple threads) fashion: all threads of a warp step through the same instruction on different data, and divergent branches are serialised. Instead of hiding memory latency with large caches and out-of-order execution as a CPU does, the GPU keeps thousands of warps resident and switches between them every cycle, so that while some wait for memory others compute. The price is that code must expose massive, regular data parallelism to run well.

The shift from graphics to general computing came in stages: programmable shaders in the early 2000s, NVIDIA's CUDA platform in 2006-2007, and the 2012 AlexNet result, trained on two consumer GPUs, which showed that deep neural networks were practical on this hardware. Since the Volta generation (2017), NVIDIA GPUs contain dedicated tensor cores that perform small matrix multiply-accumulate operations per instruction in reduced precision (FP16, BF16, and on later generations FP8 and FP4), delivering most of the chip's advertised FLOPS. Software rarely targets the hardware directly; frameworks such as PyTorch call vendor libraries (cuBLAS, cuDNN, or ROCm on AMD) and increasingly compiler-generated kernels.

Performance is best reasoned about with the roofline model: a kernel is compute-bound if its arithmetic intensity (FLOPs per byte moved from memory) exceeds the ratio of peak FLOPS to memory bandwidth, and memory-bound otherwise. Large-batch training and the prefill phase of LLM inference are mostly compute-bound; token-by-token decoding at small batch sizes is memory-bound, because every generated token requires reading all weights and the KV cache from high-bandwidth memory (HBM). This is why HBM capacity and bandwidth, not peak FLOPS, often decide which model fits on which card and how fast it answers, and why quantization and batching matter so much.

Models larger than one device are split across GPUs with tensor, pipeline, data or expert parallelism, which makes the interconnect critical: NVLink and NVSwitch within a server, and InfiniBand or RDMA-capable Ethernet between servers. Common misconceptions are that more GPUs always means proportionally more speed (communication overhead and memory limits break linear scaling) and that consumer and data-centre cards are interchangeable (they differ in memory size, interconnect, ECC and licensing terms for data-centre use). Compared with a TPU, the GPU is more general and is sold by several vendors with a broad software ecosystem, whereas the TPU is a Google-designed accelerator centred on systolic matrix units and mainly offered as a cloud service. Operationally, power and cooling are real constraints: current data-centre GPUs draw several hundred watts to over a kilowatt each, and dense racks need liquid cooling.