Dimensionality reduction

Principal component analysis (PCA), introduced by Pearson (1901) as fitting lines and planes of closest fit, is the reference linear method. scikit-learn describes it as decomposing a multivariate dataset into a set of successive orthogonal components that explain a maximum amount of the variance. In practice the data is centred (scikit-learn centres but does not scale), the singular value decomposition is computed, and each example is projected onto the top k right singular vectors; the explained variance ratio of each component guides the choice of k. Because PCA is sensitive to feature scale, features are normally standardised first. Variants include incremental and randomised PCA for large data, TruncatedSVD for sparse matrices (latent semantic analysis on text), kernel PCA for non-linear structure, and NMF when components should be non-negative.

Non-linear manifold learning methods assume the data lies near a lower-dimensional surface inside the high-dimensional space. Isomap, locally linear embedding, t-SNE (van der Maaten and Hinton, 2008) and UMAP are the common ones. t-SNE and UMAP are mainly visualisation tools: scikit-learn warns that t-SNE is stochastic, can land in local minima and does not preserve global structure, so distances between clusters and cluster sizes in a t-SNE plot should not be read literally. Autoencoders learn a non-linear compression with a neural network bottleneck, and learned embeddings from deep models are themselves a form of dimensionality reduction.

Dimensionality reduction is distinct from feature selection, which keeps a subset of the original columns rather than building new combined ones; selected features stay interpretable, while principal components are mixtures that are harder to explain. The motivation is the curse of dimensionality: as dimensions grow, data becomes sparse, distances concentrate so nearest neighbours become less meaningful, and the number of examples needed to cover the space grows exponentially.

Reduction is fitted like any other learned transform, on training data only, and then applied to held-out data. Supervised alternatives such as linear discriminant analysis use the labels to choose directions that separate classes, whereas PCA ignores labels and can discard low-variance directions that happen to be the most predictive.