Model training

The core loop is minibatch stochastic gradient descent: sample a batch, run a forward pass, compute the loss, backpropagate to get gradients, and let an optimiser update the weights. Adam and its decoupled-weight-decay variant AdamW are the de facto defaults for neural networks (Adam's usual defaults are β1 = 0.9, β2 = 0.999, ε = 1e-8), while plain SGD with momentum remains common in vision. The learning rate is typically scheduled, often with a linear warm-up followed by cosine or linear decay, and gradient-norm clipping guards against loss spikes.

For large models the engineering dominates. Mixed-precision training keeps a master copy of the weights in FP32 while doing most arithmetic in BF16 or FP16. Work is spread over many accelerators with data parallelism (each device holds a replica and gradients are averaged), tensor parallelism (single matrices split across devices), pipeline parallelism (layers split into stages) and sharded optimiser states such as ZeRO/FSDP. A common rule of thumb puts training compute for a dense transformer at roughly 6 × parameters × training tokens FLOPs. Long runs checkpoint weights and optimiser state regularly, because hardware failures are routine at cluster scale.

Modern foundation models are trained in stages: pretraining on a very large corpus with a self-supervised objective, then post-training such as supervised instruction tuning and preference optimisation (RLHF or direct methods). Fine-tuning and parameter-efficient methods such as LoRA are further training runs on top of a checkpoint, not a separate process. Training is also distinct from hyperparameter search, which is an outer loop that runs many trainings and compares them on the validation set.

Diagnostics come from the loss curves. Training loss falling while validation loss rises indicates overfitting; both staying high indicates underfitting or a bug; sudden spikes or NaNs point to an excessive learning rate or numerical instability. Reproducibility is harder than it looks: random seeds, data order, non-deterministic GPU kernels and library versions all change the result, so audit-grade training records the full configuration, data snapshot hashes and code version.

Regulation attaches to this stage. Under the EU AI Act, a general-purpose AI model is presumed to have high-impact capabilities, and so is classified as a model with systemic risk, when the cumulative compute used for its training exceeds 10^25 floating-point operations (Article 51(2) with 51(1)(a)), and providers of general-purpose models must document the training process (Article 53(1)(a)) and publish a summary of training content (Article 53(1)(d)). Security-wise, training is when data poisoning and backdoors become embedded, and the resulting weights are an asset whose theft transfers the entire training investment.