{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/deep-learning","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/deep-learning/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/deep-learning/"},"term":{"en":"Deep learning","da":"Deep learning"},"aka":{"en":[],"da":["dyb læring"]},"domain":["ai"],"cluster":"ml-fundamentals","layer":"model","status":"current","era":2012,"summary":{"en":"Machine learning that uses neural networks with many stacked layers, the approach behind modern image, speech and language tools.","da":"Maskinlæring med neurale netværk i mange lag oven på hinanden - metoden bag moderne værktøjer til billeder, tale og sprog."},"body":{"formal":{"en":"A kind of machine learning that trains neural networks with many layers, where each layer builds more general features from the output of the layer below, learned directly from raw data.","da":"En form for maskinlæring, der træner neurale netværk med mange lag, hvor hvert lag bygger mere generelle træk ud fra laget nedenunder, lært direkte fra rå data."},"plain":{"en":"Like a line of workers where the first notices edges, the next shapes, the next faces, and each passes a richer picture up the line.","da":"Som en række arbejdere, hvor den første ser kanter, den næste former og den næste ansigter - hver giver et rigere billede videre op ad rækken."},"inPractice":{"en":"A region's hospitals use a deep learning model, trained on hundreds of thousands of X-ray images, to mark possible fractures for a doctor to check.","da":"En regions hospitaler bruger en deep learning-model, der er trænet på hundredtusindvis af røntgenbilleder, til at markere mulige knoglebrud, som en læge derefter tjekker."},"whyItMatters":{"en":"It brought a large jump in what AI can do from about 2012, but the models need huge amounts of data and computing power, and their reasoning is very hard to explain.","da":"Det gav et stort spring i, hvad AI kan, fra omkring 2012, men modellerne kræver enorme mængder data og regnekraft, og deres ræsonnementer er meget svære at forklare."}},"deepDive":{"en":"The defining idea is representation learning: instead of engineers hand-crafting features (edge detectors, MFCCs for audio, n-gram counts for text) and feeding them to a shallow model, a deep network learns a hierarchy of features end to end from raw input, with all layers optimised jointly by backpropagation against a single loss. Depth matters because composing many simple nonlinear transformations can represent some functions far more compactly than a shallow network of comparable size.\n\nThe ideas are old; what changed around 2012 was the combination of large labelled datasets, GPU computation and a handful of training techniques. AlexNet (Krizhevsky, Sutskever and Hinton) won the ImageNet ILSVRC 2012 challenge with a top-5 error of about 15.3 percent against about 26.2 percent for the runner-up, trained on two consumer GPUs and using ReLU activations, dropout and data augmentation. Speech recognition had seen a similar shift a year or two earlier. ResNet (He et al., 2015) introduced residual (skip) connections, which let networks of 100+ layers train by giving gradients a direct path backwards. The transformer (Vaswani et al., 2017) then replaced recurrence with attention and became the default architecture for language, and increasingly for vision and audio.\n\nSince around 2020, empirical scaling laws have shaped practice: Kaplan et al. (2020) and Hoffmann et al. (2022, \"Chinchilla\") showed that loss falls roughly as a power law in parameters, data and compute, with Chinchilla suggesting on the order of 20 training tokens per parameter for compute-optimal training. This drove the move to foundation models trained once with self-supervision and adapted many times through fine-tuning or prompting.\n\nKnown weaknesses follow from the method. Models are data- and compute-hungry, are poorly calibrated out of distribution, are vulnerable to adversarial examples (small, crafted input perturbations that flip outputs), and can exploit spurious correlations in the training data, for example a hospital-specific marker on X-rays rather than the pathology. Explainability remains partial. Many tabular problems are still solved better by gradient-boosted trees, so \"deep\" is not automatically \"better\".\n\nTerminology: deep learning is a subset of machine learning that uses neural networks with multiple hidden layers; there is no fixed layer threshold. The neural network is the model family, while deep learning refers to the practice of training deep instances of it at scale, including the surrounding toolkit of optimisers, normalisation, regularisation and accelerator hardware.","da":"Den bærende idé er repræsentationslæring: I stedet for at ingeniører håndlaver features (kantdetektorer, MFCC'er til lyd, n-gram-optællinger til tekst) og fodrer dem til en flad model, lærer et dybt netværk et hierarki af features fra ende til anden direkte fra råt input, hvor alle lag optimeres samlet med backpropagation mod én tabsfunktion. Dybden betyder noget, fordi en sammensætning af mange simple ikke-lineære transformationer kan repræsentere visse funktioner langt mere kompakt end et fladt netværk af tilsvarende størrelse.\n\nIdéerne er gamle; det, der ændrede sig omkring 2012, var kombinationen af store mærkede datasæt, GPU-beregning og en håndfuld træningsteknikker. AlexNet (Krizhevsky, Sutskever og Hinton) vandt ImageNet-konkurrencen ILSVRC 2012 med en top-5-fejlrate på ca. 15,3 procent mod ca. 26,2 procent for nummer to, trænet på to forbruger-GPU'er og med ReLU-aktiveringer, dropout og dataaugmentering. Talegenkendelse havde set et lignende skifte et år eller to tidligere. ResNet (He m.fl., 2015) indførte residual- eller skip-forbindelser, som gør det muligt at træne netværk med over 100 lag, fordi gradienterne får en direkte vej baglæns. Transformeren (Vaswani m.fl., 2017) erstattede derefter rekurrens med attention og blev standardarkitekturen for sprog og i stigende grad for billeder og lyd.\n\nSiden omkring 2020 har empiriske skaleringslove præget praksis: Kaplan m.fl. (2020) og Hoffmann m.fl. (2022, \"Chinchilla\") viste, at tabet falder nogenlunde som en potenslov i parametre, data og regnekraft, og Chinchilla pegede på størrelsesordenen 20 træningstokens pr. parameter for beregningsoptimal træning. Det drev skiftet til foundation models, der trænes én gang med selvsupervision og tilpasses mange gange via finjustering eller prompting.\n\nDe kendte svagheder følger af metoden. Modellerne kræver store mængder data og regnekraft, er dårligt kalibrerede uden for træningsfordelingen, er sårbare over for adversarial examples (små, konstruerede ændringer af input, der vender resultatet) og kan udnytte tilfældige sammenhænge i træningsdata, fx en hospitalsspecifik markør på røntgenbilleder frem for selve sygdomstegnet. Forklarbarheden er fortsat begrænset. Mange tabelbaserede problemer løses stadig bedre med gradient-boostede træer, så \"dyb\" er ikke automatisk \"bedre\".\n\nBegrebsbrug: Deep learning er en delmængde af maskinlæring, der bruger neurale netværk med flere skjulte lag; der findes ingen fast grænse for antallet af lag. Det neurale netværk er modelfamilien, mens deep learning betegner praksis med at træne dybe udgaver af den i stor skala, inklusive den omgivende værktøjskasse af optimeringsalgoritmer, normalisering, regularisering og acceleratorhardware."},"edges":[{"type":"requires","to":"ai/neural-network","confidence":"high","strength":"normal"},{"type":"kind-of","to":"ai/machine-learning","confidence":"high","strength":"normal"},{"type":"contrasts-with","to":"ai/feature-engineering","why":{"en":"Deep learning finds useful inputs by itself from raw data; feature engineering has people build those inputs by hand.","da":"Deep learning finder selv brugbare input i rådata; ved feature engineering bygger mennesker de input i hånden."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/gpu","why":{"en":"Modern deep learning became practical because GPUs can do the huge amounts of number work it needs side by side.","da":"Moderne deep learning blev praktisk muligt, fordi GPU'er kan udføre de enorme mængder talarbejde, det kræver, side om side."},"confidence":"medium","strength":"normal"}],"depth":1,"sources":[{"title":"Goodfellow, Bengio & Courville (2016), Deep Learning","url":"https://www.deeplearningbook.org/","tier":"textbook","publisher":"MIT Press"},{"title":"Krizhevsky, Sutskever & Hinton (2012), ImageNet Classification with Deep Convolutional Neural Networks","url":"https://proceedings.neurips.cc/paper_files/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html","tier":"reference","publisher":"NeurIPS"},{"title":"He et al. (2015), Deep Residual Learning for Image Recognition","url":"https://arxiv.org/abs/1512.03385","tier":"reference"},{"title":"Hoffmann et al. (2022), Training Compute-Optimal Large Language Models","url":"https://arxiv.org/abs/2203.15556","tier":"reference"},{"title":"ISO/IEC 22989:2022, Artificial intelligence concepts and terminology","url":"https://www.iso.org/standard/74296.html","tier":"standard","publisher":"ISO/IEC"}],"draft":true}