Diffusion model

A denoising diffusion probabilistic model (DDPM; Ho, Jain and Abbeel, 2020, building on Sohl-Dickstein et al., 2015) defines a fixed forward Markov chain that adds Gaussian noise over T steps: q(x_t | x_{t−1}) = N(√(1 − β_t) x_{t−1}, β_t I). Because Gaussians compose, any step can be sampled directly as x_t = √ᾱ_t x_0 + √(1 − ᾱ_t) ε, where ᾱ_t is the running product of (1 − β_t) and ε is standard normal noise. The DDPM paper used T = 1000 with β rising linearly from 10⁻⁴ to 0.02. A network ε_θ(x_t, t) is trained to predict the added noise with a simple mean-squared error, ‖ε − ε_θ(x_t, t)‖², which is equivalent to a reweighted variational bound and, as Song et al. (2021) showed, to learning the score (gradient of the log-density) of the noised data. Sampling runs the chain backwards from pure noise, each step subtracting predicted noise and adding a little fresh noise.

Practical systems changed almost every piece. DDIM (Song, Meng and Ermon, 2020) gave a deterministic sampler that skips steps, cutting 1,000 steps to a few dozen; later ODE solvers and distillation (for example consistency models) reach one to four steps. Classifier-free guidance (Ho and Salimans) trains the network with and without the conditioning and at sampling time extrapolates, ε̃ = ε(x, ∅) + w · (ε(x, c) − ε(x, ∅)); a guidance scale w above 1 gives stronger prompt adherence at the cost of diversity and, at high values, oversaturated images. Latent diffusion (Rombach et al., 2022), the basis of Stable Diffusion, runs the process in the latent space of a variational autoencoder that downsamples images by a factor of 8 per side, with a U-Net denoiser that reads the text prompt through cross-attention to text-encoder embeddings; newer models replace the U-Net with a transformer (DiT) and often use the closely related flow-matching or rectified-flow objectives.

The contrast with autoregressive language models is in how output is formed: a diffusion model refines all pixels, audio samples or video frames jointly over many steps, so cost scales with the number of denoising steps rather than output length, and edits such as inpainting fall out naturally by fixing part of the input. Diffusion has also been applied to text, but autoregressive decoders remain dominant there.

Known problems include memorisation (Carlini et al., 2023 extracted near-copies of training images from Stable Diffusion), weak rendering of text, counts and spatial relations, and inherited bias from web-scraped training data. For provenance, output can be labelled with C2PA content credentials or invisible watermarks, and Art. 50(2) of the EU AI Act requires providers of systems that generate synthetic images, audio, video or text to mark outputs in a machine-readable, detectable way; watermarks, however, can often be removed by cropping, re-encoding or regeneration.