Tensor processing unit (TPU)

Google began deploying the first TPU in its data centres in 2015 and described it publicly in 2016; the ISCA 2017 paper by Jouppi et al. gave the details. TPU v1 was an inference-only coprocessor attached over PCIe, built around a matrix unit of 65,536 8-bit multiply-accumulate cells (a 256 × 256 array) and delivering 92 TOPS of peak integer throughput. Its central design choice was the systolic array: operands flow rhythmically through a grid of simple processing elements, each passing partial sums to its neighbour, so a large matrix multiplication proceeds with very few reads and writes to memory. This saves the energy that a general processor spends on caches, instruction fetch and register files, and it is why TPUs achieve high performance per watt on dense linear algebra while being poor at irregular, control-heavy code.

Later generations turned the chip into a training platform. TPU v2 (2017) added high-bandwidth memory, floating-point support and the bfloat16 format - 8 exponent bits like FP32 but only 7 mantissa bits - which preserves FP32's dynamic range and has since been adopted widely by other hardware. From v2 onward, chips are connected by a dedicated inter-chip interconnect (ICI) into "pods"; TPU v4 introduced optically switched 3D-torus topologies for large slices, and later generations continued with v5e and v5p, Trillium (v6e) and Ironwood (TPU7x), announced in April 2025, which Google documents with 192 GB of HBM per chip, native FP8 support and pods of up to 9,216 chips. Many generations combine TensorCores, containing the matrix units plus vector and scalar units, with SparseCores for embedding-heavy workloads such as recommendation models.

Programming TPUs goes through the XLA compiler, which traces a computation graph and compiles it for the fixed-function hardware. JAX is the most natural front end, with TensorFlow and PyTorch/XLA also supported. The compile-first model has practical consequences: static tensor shapes are strongly preferred, because changing shapes triggers recompilation; custom CUDA kernels do not port and must be rewritten (for example with Pallas) or avoided; and debugging and profiling use different tooling from the GPU world. This, rather than raw performance, is usually the main migration cost.

Compared with GPUs, TPUs trade generality and vendor choice for efficiency on neural-network workloads and tight integration with Google's infrastructure. They are primarily consumed as Google Cloud capacity and inside Google's own products, so adopting them means accepting that cloud as the execution environment, which is a lock-in and data-residency consideration. The TPU is also one instance of a broader class of AI accelerators (ASICs such as AWS Trainium and Inferentia), and it should not be confused with Google's Edge TPU, a small low-power inference chip for devices that shares the name but not the data-centre architecture.