Data labeling

Labelling covers very different annotation types: document- or image-level class labels, bounding boxes, polygons and pixel masks for vision, span and relation annotation for named-entity recognition, relevance judgements for search, transcripts for speech, and pairwise preference rankings for RLHF. Each project rests on a written guideline (a codebook) defining every label with positive and negative examples and explicit rules for edge cases, refined in pilot rounds until annotators converge. Quality control in production relies on gold items with known answers mixed into the queue, overlap where several annotators label the same item, and review or adjudication of disagreements. Open-source tools such as Label Studio and CVAT handle queues, overlap and export formats.

Inter-annotator agreement quantifies how well-defined the task is. Cohen's kappa corrects two-rater agreement for chance, Fleiss' kappa extends this to many raters, and Krippendorff's alpha handles missing ratings and ordinal or interval scales. Low agreement usually indicates an ambiguous guideline or a genuinely subjective task rather than careless annotators, and it sets a ceiling on measurable model performance. Multiple labels can be aggregated by majority vote, by an adjudicator, or by probabilistic models such as Dawid and Skene's (1979) EM approach, which estimates a confusion matrix per annotator; some teams keep the label distribution as a soft target instead of forcing a single answer.

Because labelling is expensive, much effort goes into doing less of it. Active learning queries the examples a current model is least certain about; weak supervision, as in Snorkel (Ratner et al., 2017), combines noisy programmatic labelling functions into probabilistic labels; and model-assisted pre-labelling or LLM-generated labels let humans correct rather than create. Pre-labelling speeds work but anchors annotators towards the model's suggestion, which can hide exactly the errors the project needs to find. Label errors are common even in famous data sets: Northcutt et al. (2021) estimated an average of at least 3.3% across ten benchmark test sets, and confident-learning tools such as cleanlab are used to flag likely mislabelled items for review.

Labelling is also a compliance and supply-chain activity. Under the EU AI Act, data governance for high-risk systems must cover data-preparation operations "such as annotation, labelling, cleaning" (Art. 10(2)(c)). When annotators see personal data, an external labelling vendor is a processor under GDPR Art. 28, requiring a data processing agreement and, for work done outside the EEA, a Chapter V transfer basis. Working conditions are a known risk: in 2023 TIME reported that Kenyan workers labelling toxic content for OpenAI via the contractor Sama were paid less than two dollars an hour.