Accuracy

For a binary classifier, accuracy = (TP + TN) / (TP + TN + FP + FN); for k classes it is the trace of the k × k confusion matrix divided by the total count, which makes it exactly 1 minus the empirical 0-1 loss. Variants change what counts as correct: top-k accuracy (popularised by the ImageNet top-5 metric) accepts the true class anywhere among the k highest-scoring predictions, and in multilabel problems scikit-learn's accuracy_score computes subset accuracy, where a sample only counts if every one of its labels is right, a far stricter number than per-label accuracy.

The best-known failure is the accuracy paradox. With a positive-class prevalence p, a constant classifier that always predicts the majority class scores 1 − p, so 99% accuracy is worthless at 1% prevalence. Accuracy should therefore always be reported next to the majority-class baseline. Alternatives that resist imbalance include balanced accuracy (the mean of per-class recall, (TPR + TNR) / 2 in the binary case), Cohen's kappa, which subtracts the agreement expected by chance, and the Matthews correlation coefficient, which uses all four cells and is only high when both classes are predicted well.

Accuracy is also threshold-dependent. Most models output scores or probabilities, and accuracy is measured after cutting them at a decision threshold, commonly 0.5 by library default. It is not a proper scoring rule: it ignores how confident the model was, so two models with identical accuracy can differ sharply in calibration, which log loss or the Brier score expose. It also treats every error as equally costly, which is rarely true when a missed fraud and a blocked legitimate payment have very different prices.

Being an estimate from a finite test set, it carries sampling error: the standard error is roughly √(a(1 − a)/n), so 90% measured on 1,000 cases has a 95% interval of about ±1.9 percentage points. Differences between two models on the same test cases should be checked with a paired test such as McNemar's rather than by eye. Note the naming clash with metrology, where ISO 5725-1 defines accuracy as trueness plus precision, and with the EU AI Act, whose Art. 15 uses "accuracy" broadly and requires the levels and "relevant accuracy metrics" of high-risk systems to be declared in the instructions for use (Art. 15(3)). ISO/IEC TS 4213:2022 gives a methodology for assessing classification performance, including the choice of such metrics.