Confusion matrix

For k classes the confusion matrix C is a k × k table where C[i][j] counts test items whose true class is i and whose predicted class is j, so correct predictions lie on the diagonal. Orientation is a convention, not a law: scikit-learn's confusion_matrix puts true classes in rows and predictions in columns, while some textbooks and tools transpose it, so the axes must be checked before reading a published matrix. Any class c can be reduced to a one-vs-rest 2 × 2 table: TP = C[c][c], FN = row sum minus the diagonal cell, FP = column sum minus the diagonal cell, TN = everything else.

Almost every classification metric is a function of these cells. Accuracy is the trace divided by N; recall for a class is the diagonal cell over its row sum, precision the diagonal cell over its column sum, specificity TN / (TN + FP). Row-normalising the matrix (scikit-learn normalize='true') turns each row into per-class recall; column-normalising (normalize='pred') gives per-class precision. Multiclass aggregates differ in how they combine the one-vs-rest tables: macro averaging weights every class equally, micro averaging pools the counts (and in single-label multiclass problems equals accuracy), and weighted averaging weights by class support. Cohen's kappa and the Matthews correlation coefficient are also computed directly from the matrix.

A confusion matrix describes one operating point. A model that outputs scores yields a different matrix for every threshold; the ROC curve (Fawcett, 2006) plots TPR against FPR across that family, and the precision-recall curve does the same for precision and recall. Multiplying the matrix element-wise by a cost matrix and summing gives the expected cost of a threshold, which is how cost-sensitive decisions are made explicit. Because precision depends on prevalence while TPR and FPR do not, a matrix measured on a deliberately balanced test set cannot be read as production performance without reweighting to the real class mix.

In error analysis the off-diagonal hot spots are the useful part: systematic confusion between two classes often signals overlapping definitions in the labelling guideline, ambiguous source data or a taxonomy that should merge or split classes, rather than a model defect. Computing separate matrices per population slice supports fairness checks such as equalised odds (Hardt et al., 2016), which compares TPR and FPR across groups. For multilabel tasks, scikit-learn's multilabel_confusion_matrix returns one 2 × 2 table per label. Small cells are statistically noisy, so rare classes need enough test examples before their row can be trusted.