Epoch

With N training examples and batch size B, an epoch consists of ⌈N/B⌉ optimiser steps, or ⌊N/B⌋ when the last incomplete batch is dropped (drop_last=True in a PyTorch DataLoader). The data is normally reshuffled at the start of each epoch, so training samples without replacement within an epoch; this random reshuffling generally works better in practice than the with-replacement sampling assumed in much SGD theory. With on-the-fly data augmentation each epoch sees different random crops, flips or noise, so later epochs are not exact repeats. In distributed training each replica must see a disjoint shard per epoch, which is why samplers such as PyTorch's DistributedSampler need set_epoch() called each epoch, otherwise every epoch repeats the same order.

The epoch is a convenient unit for logging, checkpointing and validation, but it is not what the optimiser sees; learning-rate warmup and cosine or step decay are defined in steps, so changing the batch size silently changes how many steps an epoch-based schedule contains. Typical counts vary enormously: classic ImageNet ResNet-50 recipes use about 90 epochs, small tabular or vision data sets may use hundreds, and fine-tuning a pretrained language model usually uses only one to three epochs, since more tends to cause memorisation and loss of general ability.

Large-scale pretraining has largely abandoned multi-epoch training over the whole corpus. Most web data is seen once or less, while smaller high-quality sources are upsampled and repeated. Muennighoff et al. (2023) found that, for a fixed compute budget, up to about four epochs of repeated data yield almost the same loss as unique data, after which the value of repetition falls off quickly. Repetition also has privacy implications: sequences duplicated many times in training data are far more likely to be memorised and regurgitated verbatim (Carlini et al., 2022), which is one reason deduplication is a standard preprocessing step.

The number of epochs is usually chosen by early stopping on the validation set, but the loss curve does not always behave as the textbook U-shape suggests. Nakkiran et al. (2019) documented epoch-wise double descent, where test error rises and then falls again with further training in sufficiently large models, and Power et al. (2022) described "grokking", where small networks on algorithmic tasks generalise long after fitting their training set perfectly. Patience-based stopping is therefore a heuristic, and for expensive runs it pays to keep periodic checkpoints rather than only the best one.