Model parameter

In a neural network the parameters are every tensor that the optimiser updates: weight matrices and bias vectors of linear layers, convolution kernels, token-embedding tables, learned positional embeddings and the scale and shift (γ, β) of normalisation layers. Not every stored number is a parameter. Batch normalisation's running mean and variance are updated by averaging, not by gradients, and PyTorch therefore registers them as buffers rather than parameters; the optimiser's own state, such as Adam's moment estimates, is not part of the model either, although it is saved in training checkpoints.

Counting is mechanical. A dense layer mapping d_in to d_out has d_in·d_out + d_out parameters. In a standard transformer block with model width d and a 4× MLP expansion, attention contributes about 4d² and the MLP about 8d², so the non-embedding parameter count is approximately 12·L·d² for L layers (Kaplan et al., 2020); GPT-3's 96 layers at d = 12,288 give about 174 billion, matching its advertised 175B once embeddings are added. The embedding table adds vocabulary size × d, and is often tied to the output layer. Mixture-of-experts models distinguish total from active parameters: Mixtral 8x7B has about 47 billion parameters in total but uses roughly 13 billion per token, so its memory footprint and its per-token compute tell different stories.

Memory follows from parameter count and numeric format: 4 bytes per parameter in FP32, 2 in BF16 or FP16, 1 in INT8 and about 0.5 at 4 bits. A 7-billion-parameter model therefore needs about 14 GB for its weights in BF16, before the KV cache and activations. Training is far heavier: mixed-precision training with Adam needs roughly 16 bytes per parameter for weights, gradients, FP32 master weights and two moment estimates, as analysed in the ZeRO paper, so a 7B model needs over 100 GB for model and optimiser state alone.

Parameter count is a poor proxy for capability on its own. Kaplan et al. (2020) found loss falling as a power law in parameters, but Hoffmann et al. (2022) showed that many large models were undertrained and that compute-optimal training uses on the order of 20 tokens per parameter; their 70B Chinchilla, trained on 1.4 trillion tokens, outperformed the 280B Gopher. Training compute is commonly estimated as C ≈ 6·N·D for N parameters and D tokens. Regulation reflects this: the EU AI Act presumes that a general-purpose model has high-impact capabilities, the trigger for classing it as a model with systemic risk, when its cumulative training compute exceeds 10²⁵ floating-point operations (Art. 51(2)); the test is compute, not parameter count. Contrast hyperparameters, which are chosen, not learned, and weights, which are the largest subset of parameters.