AI bias

NIST SP 1270 (2022) sorts AI bias into three interacting categories: systemic bias embedded in institutions and historical practice, statistical and computational bias arising from non-representative samples and modelling choices, and human bias in how people design, label, interpret and act on outputs. The technical entry points are well known: sampling and representation bias (a group is rare in the training set), label bias (the target encodes past decisions, such as "was hired" or "was investigated", rather than the ground truth), measurement bias (a feature is recorded less accurately for some groups), aggregation bias (one model for populations that behave differently) and deployment bias (a model used outside the context it was validated for).

Removing a protected attribute does not remove the bias. Postcode, name, language, school or employment gaps act as proxies, and a flexible model will reconstruct the missing attribute from them. For this reason audits measure outcomes rather than inspect inputs. Common group metrics are demographic parity (equal selection rates), the US "four-fifths" rule of thumb for adverse impact (a selection-rate ratio below 0.8), equalised odds (equal true- and false-positive rates, Hardt et al., 2016) and calibration within groups. Kleinberg et al. and Chouldechova (both 2016-17) proved that when base rates differ, calibration and equal error rates cannot hold at once except for a perfect predictor - the core of the ProPublica-Northpointe dispute over the COMPAS recidivism score. Choosing a metric is therefore a normative decision that must be documented, not a purely technical one.

Mitigation happens at three points: pre-processing (re-sampling, re-weighting, relabelling), in-processing (fairness constraints or adversarial debiasing in the loss) and post-processing (group-specific thresholds). Each trades some accuracy or individual consistency for group parity, and group-specific thresholds may themselves raise equal-treatment questions. For generative models, bias shows up as stereotyped text or images and uneven refusal rates, and is measured with benchmark prompt sets and counterfactual prompts that swap only a name or pronoun.

In EU law the EU AI Act Art. 10(2)(f)-(g) requires providers of high-risk systems to examine training, validation and test data for possible biases and to take measures to detect, prevent and mitigate them, and Art. 10(5) permits processing of special-category personal data strictly for that purpose under safeguards; the 2026 Digital Omnibus amendment (Regulation (EU) 2026/1744) widened that legal basis beyond high-risk systems. GDPR Art. 22 and the principle of fairness in Art. 5(1)(a) apply in parallel, as do national equal-treatment laws. Danish public-sector profiling has drawn scrutiny, for example Amnesty International's 2024 report on Udbetaling Danmark's fraud-control algorithms. Bias differs from hallucination (a fabricated output rather than a systematic tilt) and from data poisoning, where the skew is introduced deliberately by an attacker.