Activation function

A unit computes a = phi(w . x + b). If phi is linear, a stack of layers is a composition of affine maps and therefore itself affine, so depth adds no expressive power. With a suitable non-polynomial phi, a network with one hidden layer is a universal approximator of continuous functions on compact sets (Cybenko, 1989; Hornik, 1991; Leshno et al., 1993), and depth makes many functions far cheaper to represent. The activation also sets the gradient that backpropagation multiplies through each layer, which is why its derivative matters as much as its shape.

The logistic sigmoid 1/(1 + e^-x) and tanh dominated early networks. Both saturate: for large positive or negative inputs their derivative is close to zero (the sigmoid's derivative is at most 0.25), so gradients shrink geometrically as they flow back through many layers, the vanishing gradient problem. The rectified linear unit, ReLU(x) = max(0, x), was popularised by Nair and Hinton (2010) and by Glorot, Bordes and Bengio (2011), who showed deep rectifier networks could train well without unsupervised pretraining. ReLU has derivative 1 for positive inputs and gives sparse activations, but units stuck at negative inputs output zero forever (dying ReLU), which led to Leaky ReLU, PReLU and ELU.

Modern transformers mostly use smooth gated variants. GELU (Hendrycks and Gimpel, 2016) is x times the standard normal CDF of x and is used in BERT and GPT-2; SiLU or Swish is x times sigmoid(x); SwiGLU, a gated linear unit with a Swish gate, is used in the feed-forward blocks of models such as PaLM and LLaMA. Output layers use task-specific functions rather than hidden-layer activations: sigmoid for independent binary outputs, softmax for a probability distribution over classes or tokens, and identity for regression. Activation choice interacts with weight initialisation: Glorot (Xavier) initialisation suits tanh, while He initialisation is derived for ReLU. PyTorch provides these as torch.nn modules such as ReLU, GELU, SiLU, Sigmoid, Tanh and Softmax.