Pretraining

Pretraining objectives are self-supervised: the targets are derived from the input itself. Decoder-only language models, following GPT (Radford et al., 2018), use causal language modelling, predicting each token from all previous ones with a cross-entropy loss. BERT (Devlin et al., 2018) used masked language modelling, hiding about 15% of tokens and predicting them from both sides, which suits encoders for classification and embeddings but not free generation. T5 (Raffel et al., 2020) used span corruption, replacing contiguous spans with sentinel tokens for an encoder-decoder model to reconstruct. Text is first split into subword tokens, typically with byte-pair encoding, and the tokenizer is fixed for the life of the model.

Data engineering dominates the quality of the result. A typical pipeline starts from Common Crawl snapshots, extracts text from HTML, identifies language, applies heuristic and model-based quality filters, removes exact and near-duplicates (often with MinHash), strips or masks personal data and filters toxic content, then mixes the web data with code, books, scientific papers and encyclopaedic text at tuned proportions. FineWeb (Penedo et al., 2024) is a documented example of such a pipeline yielding about 15 trillion tokens, and Meta reported pretraining Llama 3 on over 15 trillion tokens. Late in training many recipes anneal on smaller high-quality or domain-specific data and extend the context length, a phase sometimes called mid-training.

Compute is roughly C ≈ 6·N·D floating-point operations for N parameters and D tokens. Hoffmann et al. (2022) found compute-optimal training at about 20 tokens per parameter, but current models are deliberately trained far beyond that because a smaller model that has seen more data is cheaper to serve; Llama 3 8B saw over 15 trillion tokens, nearly 2,000 per parameter. Runs use AdamW with warmup and cosine decay, BF16 mixed precision and combined data, tensor and pipeline parallelism across thousands of accelerators for weeks or months, with frequent checkpoints to recover from hardware failures and loss spikes.

The outcome is a base model that continues text, reflects the statistics and gaps of its corpus, has a knowledge cutoff set by its data and can memorise and regurgitate rare or duplicated sequences. Web-scale collection is exposed to poisoning: Carlini et al. (2023) showed that buying expired domains referenced by LAION-400M would have let an attacker control about 0.01% of it for around 60 US dollars. Since 2 August 2025 providers of general-purpose AI models in the EU must maintain a policy to comply with Union copyright law, including machine-readable text-and-data-mining opt-outs under Art. 4(3) of Directive (EU) 2019/790, and publish a sufficiently detailed summary of training content (AI Act Art. 53(1)(c) and (d)). Pretraining differs from fine-tuning and instruction tuning in scale, in using unlabelled data and in being where nearly all knowledge enters the model.