Neural network

Each unit computes y = f(w·x + b): a weighted sum of its inputs plus a bias, passed through a nonlinear activation function f. A layer applies this to many units at once, which reduces to a matrix multiplication followed by an elementwise nonlinearity, and a network composes layers. Without the nonlinearity any stack of layers would collapse into a single linear map. Common activations are the sigmoid and tanh (historically), ReLU, max(0, x), which became standard in the early 2010s because it does not saturate for positive inputs, and smooth variants such as GELU used in transformers.

Rosenblatt's perceptron (1958) was a single trainable layer with a threshold output. Minsky and Papert's 1969 analysis showed that a single layer cannot represent functions that are not linearly separable, such as XOR, which dampened interest for over a decade. Multilayer perceptrons with hidden layers remove that limit, and the universal approximation theorems (Cybenko 1989 for sigmoids, Hornik 1991 more generally) show that a single hidden layer of sufficient width can approximate any continuous function on a compact domain to arbitrary precision. The theorem guarantees existence only; it says nothing about how many units are needed or whether training will find the weights.

Training uses backpropagation, popularised by Rumelhart, Hinton and Williams in 1986, which applies the chain rule to compute the gradient of the loss with respect to every weight in one backward sweep, followed by gradient descent or a variant such as Adam. Practical difficulties include vanishing and exploding gradients in deep stacks, addressed with careful initialisation (Glorot/Xavier 2010, He 2015), normalisation layers and residual connections, and sensitivity to learning rate and batch size.

Architectures specialise the basic pattern through weight sharing and connectivity: convolutional networks share filters across image positions, recurrent networks and LSTMs (Hochreiter and Schmidhuber 1997) share weights across time steps, and transformers (2017) use attention to let every position weigh every other. Graph neural networks pass messages along edges. Parameter counts range from thousands in embedded classifiers to hundreds of billions in large language models.

The brain analogy is loose. Artificial units are differentiable arithmetic, not spiking neurons, and backpropagation has no established biological counterpart. The practical consequence for audit is that the learned function is distributed across the weight matrices; explainability methods such as saliency maps, SHAP or probing give partial, approximate views rather than the rules a reviewer might expect.