Logistic regression

Despite its name, logistic regression is a classifier. For binary classes it models p(y = 1 | x) = sigma(w0 + w^T x), where sigma(z) = 1 / (1 + e^-z) is the logistic (sigmoid) function. Equivalently, the log-odds log(p / (1 - p)) are a linear function of the features, so each coefficient is the change in log-odds per unit of its feature and e^w is an odds ratio. The decision boundary p = 0.5 is a hyperplane, so the model is linear in feature space; non-linear boundaries need transformed or engineered features.

The weights are fitted by maximum likelihood, which is the same as minimising the log-loss (binary cross-entropy) -[y log p + (1 - y) log(1 - p)]. Unlike least squares there is no closed-form solution, but the loss is convex, so iterative solvers such as Newton methods (iteratively reweighted least squares), L-BFGS or stochastic gradient descent reach the global optimum. If the classes are perfectly separable the unregularised weights grow without bound, which is one reason libraries apply regularisation by default: scikit-learn's LogisticRegression uses an L2 penalty with strength controlled by C (default 1.0), and also supports L1 and elastic net.

For more than two classes the multinomial (softmax) form gives each class its own weight vector and normalises the scores into a probability distribution; a one-vs-rest scheme of separate binary models is the older alternative. A single-layer neural network with a sigmoid or softmax output trained on cross-entropy is exactly logistic regression, which is why it is often taught as the bridge from classical statistics to deep learning.

The model's probabilities are usually reasonably well calibrated, a property that matters for credit scoring and clinical risk scores, but it still needs a chosen threshold to turn a probability into a decision, and the right threshold depends on the relative cost of false positives and false negatives. The method was developed in statistics, notably by Cox (1958), well before machine learning adopted it.