Quantization

The basic scheme is affine (uniform) quantization: a real value x is mapped to an integer q = clamp(round(x / s) + z, q_min, q_max) and approximately recovered as x̂ = s · (q − z), where s is the scale and z the zero point. Symmetric quantization fixes z = 0 and suits roughly zero-centred weights; asymmetric quantization spends the extra parameter on skewed ranges such as post-ReLU activations. Granularity matters as much as bit width: one scale per tensor is cheap but coarse, per-channel scales are standard for weights, and LLM weight quantization at 4 bits usually uses per-group scales (for example one scale per 128 consecutive weights), which adds a small storage overhead but sharply reduces error.

There are two families of methods. Post-training quantization (PTQ) converts an already trained model, sometimes using a small calibration set to choose ranges. Quantization-aware training (QAT) simulates rounding during training or fine-tuning, using a straight-through estimator for gradients, and recovers more accuracy at low bit widths at the cost of a training run. Large language models brought specific problems: LLM.int8() (Dettmers et al., 2022) showed that a few activation feature dimensions contain large outliers that break naive 8-bit schemes, and handled them in 16-bit; SmoothQuant migrates activation outliers into the weights with a per-channel rescaling; GPTQ quantizes weights layer by layer using approximate second-order information; AWQ protects the weight channels that matter most for activations. QLoRA introduced the 4-bit NormalFloat (NF4) data type for fine-tuning on quantized base models.

It is important to be clear about what is quantized. Weight-only schemes (W4A16, W8A16) store weights compactly but dequantize them for 16-bit arithmetic; they mainly help memory-bound decoding, where speed tracks bytes read per token. Weight-and-activation schemes (W8A8 in INT8 or FP8) also use low-precision tensor-core math and help compute-bound prefill. FP8 comes in two variants, E4M3 and E5M2, trading mantissa precision against range, and newer hardware adds 4-bit floating-point formats. The KV cache can be quantized separately. In the llama.cpp ecosystem, GGUF files carry block-wise formats such as Q4_K_M and Q8_0.

The memory arithmetic is simple: 70 billion parameters take about 140 GB at 16 bits, 70 GB at 8 bits and roughly 35-40 GB at 4 bits including scales, which can decide whether a model fits on one accelerator. The failure modes are less obvious. Degradation is uneven: perplexity and general benchmarks may barely move while arithmetic, code, long-context retrieval or languages other than English can degrade noticeably, so evaluation should use task-specific data. Very low bit widths (3 bits and below) usually need QAT or careful methods. Quantization differs from distillation, which trains a new smaller model, and from pruning, which removes parameters; the three are often combined.