Hyperparameter

Hyperparameters fall into several groups. Optimisation hyperparameters include the learning rate and its schedule (warmup length, decay shape), batch size, number of epochs or steps, momentum or Adam's β values, weight decay and gradient-clipping threshold. Architectural ones fix capacity: number of layers, hidden width, attention heads, dropout rate, context length. Data-side choices such as augmentation strength or the mixture weights of pretraining sources behave the same way, as do method-specific settings such as LoRA's rank r and scaling α or the KL coefficient in RLHF. Decoding settings such as temperature and top-p are sometimes also called hyperparameters, although they act at inference, not training. The boundary with learned parameters is a design choice: a temperature or a regularisation weight can be made learnable, at which point it stops being a hyperparameter.

Not all hyperparameters matter equally, and the learning rate is usually the most sensitive (Goodfellow et al., §11.4). Grid search scales exponentially with the number of dimensions and wastes trials on unimportant ones; Bergstra and Bengio (2012) showed that random search finds equally good or better configurations with far fewer trials for exactly that reason. Scale-type hyperparameters should be searched on a log scale, for example learning rates from 10⁻⁵ to 10⁻¹. Bayesian optimisation fits a surrogate model of validation score against configuration (Gaussian processes (Snoek et al., 2012) or the tree-structured Parzen estimator used by default in Optuna) and chooses the next trial by an acquisition function. Multi-fidelity methods such as successive halving and Hyperband (Li et al., 2018) stop unpromising runs early, and population-based training (Jaderberg et al., 2017) mutates hyperparameters of running jobs.

Large models cannot be tuned by brute force, since a single run may cost millions. Practitioners tune small proxy models and extrapolate, either through empirical scaling fits for learning rate and batch size or through parametrisations such as μP (Yang et al., 2022), under which optimal hyperparameters stay approximately stable as width grows, allowing "tune small, transfer large".

Hyperparameter search is also a source of unfair comparisons and silent overfitting. Every configuration tried is a look at the validation set, so the chosen score is optimistically biased and the final number must come from an untouched test set or from nested cross-validation. Comparisons between methods are only fair at equal tuning budgets; Melis et al. (2018) found that carefully tuned LSTM language models outperformed several newer architectures that had been credited with gains. The random seed behaves like a hidden hyperparameter, so results should be reported over several seeds, and every run's configuration should be logged for reproducibility.