Self-supervised learning

The approach defines a pretext task whose target can be computed from the raw input, trains a network on it with an ordinary supervised loss, and then reuses the learned representations for downstream tasks. Two families dominate. Generative or predictive objectives reconstruct hidden parts of the input: causal language modelling predicts the next token from the preceding ones (the GPT line), masked language modelling predicts hidden tokens from both sides, and masked autoencoders for images (He et al., 2021) hide a large share of image patches, typically 75 percent, and reconstruct the pixels. Contrastive and joint-embedding objectives instead learn to map different views of the same item close together and different items apart: SimCLR (2020) contrasts two augmented crops of an image, and CLIP (2021) aligns images with their captions using a contrastive loss over large batches.

BERT (Devlin et al., 2019) illustrates the details. It selects 15 percent of input tokens for prediction; of those, 80 percent are replaced by a [MASK] token, 10 percent by a random token and 10 percent are left unchanged, so the model cannot rely on seeing [MASK] at fine-tuning time. Speech works similarly: wav2vec 2.0 (Baevski et al., 2020) masks spans of latent speech features and solves a contrastive task over quantised targets, and showed that fine-tuning on as little as ten minutes of transcribed audio, after pretraining on 53,000 hours of unlabelled speech, could produce usable recognisers, which is the pattern behind low-resource languages and domain-specific dictation.

A known failure mode for joint-embedding methods is representational collapse, where the network maps every input to the same vector and trivially satisfies the objective. Negative pairs (contrastive methods), asymmetric architectures with stop-gradient (BYOL, SimSiam) or explicit variance regularisation (VICReg) prevent it. Augmentation choices also encode assumptions: cropping and colour jitter teach invariance to those changes, which is harmful if colour matters for the downstream task.

The terminology is contested. Some authors, including Yann LeCun, who popularised the term, distinguish it sharply from unsupervised learning; older literature files the same methods under unsupervised learning. The practical distinction is that self-supervised methods have an explicit prediction target and a supervised-style loss.

Scale is the consequence and the risk. Because no human labelling is needed, pretraining corpora can include trillions of tokens of web text, which is what makes foundation models possible but also means that toxic content, personal data, copyrighted material and deliberately poisoned pages enter the model unless filtered. The pretrained model is usually followed by supervised fine-tuning and preference training before deployment.