Explainability

The literature separates interpretability - models whose structure a person can inspect directly, such as sparse linear models, shallow decision trees, rule lists or generalised additive models - from post-hoc explainability, where a separate method approximates why an opaque model produced an output. Explanations are further classified as global (how the model behaves overall) or local (why this one prediction), and as model-specific or model-agnostic. Rudin (2019) argued that for high-stakes tabular decisions an interpretable model often matches black-box accuracy, making post-hoc explanation of a black box the weaker choice.

The main post-hoc families are feature attribution, example-based and counterfactual methods. LIME (Ribeiro et al., 2016) fits a weighted linear surrogate around the instance using perturbed samples. SHAP (Lundberg and Lee, 2017) assigns each feature its Shapley value from cooperative game theory, the unique attribution satisfying efficiency, symmetry, dummy and additivity; KernelSHAP estimates it by sampling and TreeSHAP computes it exactly for tree ensembles. Gradient methods for neural networks include saliency maps, Integrated Gradients (Sundararajan et al., 2017), which integrates gradients along a path from a baseline, and Grad-CAM for convolutional networks. Counterfactual explanations (Wachter et al., 2017) state the smallest change to the input that would flip the outcome - "had declared income been above X, the application would have been approved" - which is often the most useful form for an affected person.

Failure modes are well documented. Attributions depend on the chosen baseline or background distribution; correlated features split credit arbitrarily; saliency maps can look plausible while being insensitive to the model's weights (Adebayo et al., 2018, "Sanity Checks for Saliency Maps"); and LIME and SHAP can be manipulated so that a biased model appears to rely on innocuous features. For LLMs, a chain-of-thought is generated text and is not guaranteed to be a faithful account of the computation; mechanistic interpretability (circuits, probing, sparse autoencoders over activations) aims at faithful explanations but is still a research field. NIST IR 8312 accordingly lists four principles: explanation, meaningful, explanation accuracy and knowledge limits.

Legally, GDPR Arts. 13(2)(f), 14(2)(g) and 15(1)(h) give data subjects a right to meaningful information about the logic involved in automated decisions under Art. 22. The CJEU held in SCHUFA (C-634/21, 2023) that a credit score can itself be such a decision, and in Dun & Bradstreet Austria (C-203/22, 2025) that the explanation must enable the person to understand the procedure and principles actually applied, without being defeated wholesale by trade-secret claims. The EU AI Act adds Art. 13 (instructions enabling deployers to interpret output), Art. 14 (human oversight) and Art. 86, a right for affected persons to obtain clear and meaningful explanations of certain decisions based on Annex III high-risk systems.