Low-rank adaptation (LoRA)

For a frozen weight matrix W₀ ∈ ℝ^(d×k), LoRA learns an update ΔW = BA with B ∈ ℝ^(d×r), A ∈ ℝ^(r×k) and rank r ≪ min(d, k), so the layer computes h = W₀x + (α/r)·BAx. A is initialised randomly and B with zeros, so training starts exactly at the pretrained model. The parameter saving is large: for a 4096 × 4096 projection, r = 8 gives 8 × (4096 + 4096) = 65,536 trainable values instead of about 16.8 million, roughly 0.4%. Gradients and optimiser state are needed only for A and B, which is where most of the memory saving comes from; the frozen base still has to be held in memory for the forward and backward passes.

Hu et al. (2021) applied LoRA to the attention query and value projections of GPT-3 175B and reported about 10,000 times fewer trainable parameters and a threefold reduction in GPU memory compared with full fine-tuning, with quality on par. Because BA has the same shape as W₀, it can be merged into the base weights after training, adding no inference latency; alternatively adapters stay separate so many task-specific adapters can share one base model, which serving systems exploit by batching requests for different adapters together. The method rests on the observation that fine-tuning updates have low intrinsic rank. Current practice often targets all linear layers, with ranks from about 4 to 64, using libraries such as Hugging Face PEFT.

QLoRA (Dettmers et al., 2023) trains LoRA adapters on top of a base model quantised to 4-bit NormalFloat, adding double quantisation of the quantisation constants and paged optimiser states, which made fine-tuning a 65-billion-parameter model possible on a single 48 GB GPU. Other variants include DoRA, which separates the magnitude and direction of the weight update, and rank-stabilised LoRA, which scales by α/√r instead of α/r so higher ranks remain effective.

LoRA is not a free replacement for full fine-tuning. Biderman et al. (2024) found that in continued training on code and mathematics it substantially underperforms full fine-tuning, whose weight updates had 10-100 times higher rank than typical LoRA settings, but that it also forgets less of the base model's other abilities. An adapter is bound to the exact base checkpoint it was trained on; loading it onto a different version or quantisation silently changes behaviour. Adapters are also a supply-chain artefact: OWASP's Top 10 for LLM Applications 2025 (LLM03) warns that a malicious LoRA adapter can compromise the integrity of a trusted base model, and research has shown that LoRA fine-tuning can cheaply strip safety training from aligned models, so third-party adapters need provenance checks and evaluation before use.