{"licence":{"name":"CC BY-SA 4.0","spdx":"CC-BY-SA-4.0","url":"https://creativecommons.org/licenses/by-sa/4.0/","attribution":"Atlas, a bilingual technical dictionary (https://cmaintz.github.io/tech-atlas/)"},"id":"ai/mixture-of-experts","url":{"en":"https://cmaintz.github.io/tech-atlas/en/terms/ai/mixture-of-experts/","da":"https://cmaintz.github.io/tech-atlas/da/terms/ai/mixture-of-experts/"},"term":{"en":"Mixture of experts (MoE)","da":"Mixture of experts (MoE)"},"aka":{"en":["MoE"],"da":["MoE"]},"domain":["ai"],"cluster":"model-architecture","layer":"model","status":"current","era":1991,"summary":{"en":"A way to build a very large model where only a few of its many parts do the work for each token, so answers cost far less.","da":"En måde at bygge en meget stor model på, hvor kun få af dens mange dele arbejder for hvert token, så svar koster langt mindre."},"body":{"formal":{"en":"A neural network split into many parallel sub-networks (“experts”) plus a small chooser that sends each token to just a few of them; the idea dates from 1991, and the modern large-scale form from 2017.","da":"Et neuralt netværk delt i mange parallelle delnetværk (“eksperter”) plus en lille vælger, der sender hvert token til kun nogle få af dem; idéen er fra 1991, og den moderne storskala-form fra 2017."},"plain":{"en":"Like a hospital full of specialists where the front desk sends each patient to the two doctors who fit best, instead of every doctor seeing every patient.","da":"Som et hospital fuldt af specialister, hvor skranken sender hver patient til de to læger, der passer bedst, i stedet for at alle læger ser alle patienter."},"inPractice":{"en":"An IT architect in a ministry compares two models of the same total size; the mixture-of-experts one answers several times faster, yet needs just as many GPUs, because every expert must be held in memory.","da":"En IT-arkitekt i et ministerium sammenligner to modeller af samme samlede størrelse; den med mixture of experts svarer flere gange hurtigere, men kræver lige så mange GPU'er, fordi alle eksperter skal ligge i hukommelsen."},"whyItMatters":{"en":"It lets builders add knowledge without the running cost growing at the same rate, which is why many of today's largest models use it; the hardware bill for memory stays large.","da":"Den lader udviklere tilføje viden, uden at driftsprisen vokser i samme takt, og derfor bruger mange af nutidens største modeller den; regningen for hukommelse forbliver dog stor."}},"deepDive":{"en":"In a sparse MoE layer a router, usually a single linear projection W_g, scores each token's hidden state x against N experts; the top k scores are kept and renormalised with a softmax, and the layer output is y = Σ g_i(x) · E_i(x) over the selected experts only. In transformer language models the experts are copies of the feed-forward (MLP) sublayer, while attention remains dense and shared. Routing is decided independently per token and per layer, so a single sentence passes through many different expert combinations. The original idea (Jacobs, Jordan, Nowlan and Hinton, 1991) used a gating network to blend a few whole models; Shazeer et al. (2017) made it sparse and conditional with noisy top-k gating between LSTM layers, reaching 137 billion parameters.\n\nThe main engineering problem is load balancing. Without a counter-pressure the router collapses onto a few experts that then get all the training signal. GShard (Lepikhin et al., 2020) used top-2 routing with an auxiliary balancing loss; Switch Transformer (Fedus, Zoph and Shazeer, 2022) simplified to top-1 routing, defined an expert capacity (tokens per expert per batch, scaled by a capacity factor) beyond which tokens are dropped and passed on via the residual path, and trained models up to 1.6 trillion parameters. ST-MoE added a router z-loss for numerical stability. Later designs use many small fine-grained experts plus one or more always-active shared experts, and DeepSeek-V3 (671 billion total parameters, about 37 billion activated per token) balances load with a per-expert bias term instead of a large auxiliary loss.\n\nThe accounting distinction is between total and active parameters. Mixtral 8x7B has 8 experts per layer with top-2 routing, 46.7 billion total parameters and 12.9 billion used per token; its compute per token resembles a 13-billion dense model, but at 16-bit precision all weights still need roughly 94 GB of accelerator memory. At small batch sizes decoding is memory-bandwidth-bound and only active experts are read, so MoE is fast; at large batch sizes different tokens hit different experts, most weights are read anyway, and the advantage shrinks. Across many devices, expert parallelism places experts on different GPUs and requires all-to-all communication of tokens in every MoE layer, which makes interconnect bandwidth a bottleneck.\n\nA frequent misconception is that experts are topical specialists, one for law and one for medicine. Analyses such as the Mixtral paper found little domain-level specialisation; routing correlates more with token types and syntax. MoE models have also been reported to be harder to fine-tune stably and more prone to overfitting on small datasets than dense models of similar active size, and routing adds a source of non-determinism when capacity limits drop tokens depending on what else is in the batch.","da":"I et sparse MoE-lag scorer en router, som regel én lineær projektion W_g, hvert tokens skjulte tilstand x mod N eksperter; de k højeste scorer beholdes og normaliseres igen med softmax, og lagets output er y = Σ g_i(x) · E_i(x) over kun de valgte eksperter. I transformer-sprogmodeller er eksperterne kopier af feed-forward-dellaget (MLP), mens attention forbliver tæt og fælles. Routingen afgøres uafhængigt for hvert token og hvert lag, så en enkelt sætning passerer gennem mange forskellige kombinationer af eksperter. Den oprindelige idé (Jacobs, Jordan, Nowlan og Hinton, 1991) brugte et gating-netværk til at blande nogle få hele modeller; Shazeer m.fl. (2017) gjorde det sparsomt og betinget med noisy top-k gating mellem LSTM-lag og nåede 137 milliarder parametre.\n\nDet største ingeniørproblem er lastbalancering. Uden et modtryk kollapser routeren ned på nogle få eksperter, som så får hele træningssignalet. GShard (Lepikhin m.fl., 2020) brugte top-2-routing med et ekstra balanceringstab; Switch Transformer (Fedus, Zoph og Shazeer, 2022) forenklede til top-1-routing, definerede en ekspertkapacitet (tokens pr. ekspert pr. batch, skaleret med en capacity factor), ud over hvilken tokens droppes og kun føres videre via residualforbindelsen, og trænede modeller på op til 1,6 billioner parametre. ST-MoE tilføjede et router z-loss for numerisk stabilitet. Nyere designs bruger mange små, finkornede eksperter plus en eller flere altid aktive fælles eksperter, og DeepSeek-V3 (671 milliarder parametre i alt, omkring 37 milliarder aktiveret pr. token) balancerer lasten med et bias-led pr. ekspert i stedet for et stort ekstra tab.\n\nRegnskabet skelner mellem samlede og aktive parametre. Mixtral 8x7B har 8 eksperter pr. lag med top-2-routing, 46,7 milliarder parametre i alt og 12,9 milliarder brugt pr. token; beregningen pr. token ligner en tæt model på 13 milliarder, men ved 16-bit-præcision kræver alle vægte stadig omkring 94 GB acceleratorhukommelse. Ved små batchstørrelser er decoding begrænset af hukommelsesbåndbredden, og kun de aktive eksperter læses, så MoE er hurtig; ved store batches rammer forskellige tokens forskellige eksperter, de fleste vægte læses alligevel, og fordelen skrumper. På tværs af mange enheder placerer ekspertparallelisme eksperterne på forskellige GPU'er og kræver all-to-all-kommunikation af tokens i hvert MoE-lag, hvilket gør båndbredden i forbindelserne mellem dem til en flaskehals.\n\nEn udbredt misforståelse er, at eksperterne er faglige specialister, én til jura og én til medicin. Analyser som Mixtral-artiklen fandt kun ringe specialisering på domæneniveau; routingen hænger mere sammen med tokentyper og syntaks. MoE-modeller er også rapporteret at være sværere at finjustere stabilt og mere tilbøjelige til overfitting på små datasæt end tætte modeller af tilsvarende aktiv størrelse, og routing tilføjer en kilde til ikke-determinisme, når kapacitetsgrænser dropper tokens afhængigt af, hvad der ellers er i batchen."},"edges":[{"type":"requires","to":"ai/model-parameter","why":{"en":"The design depends on the gap between how many model parameters exist in total and how many are used for each token.","da":"Designet bygger på forskellen mellem, hvor mange modelparametre der findes i alt, og hvor mange der bruges for hvert token."},"confidence":"high","strength":"primary"},{"type":"kind-of","to":"ai/neural-network","confidence":"high","strength":"normal"},{"type":"used-with","to":"ai/transformer","why":{"en":"In today's large language models, the expert parts usually replace one block inside each transformer layer.","da":"I nutidens store sprogmodeller erstatter ekspertdelene som regel én blok inde i hvert transformer-lag."},"confidence":"high","strength":"primary"},{"type":"used-with","to":"ai/model-serving","why":{"en":"Serving it is tricky because every expert must sit in memory even though only a few run at a time.","da":"Driften er vanskelig, fordi alle eksperter skal ligge i hukommelsen, selvom kun få kører ad gangen."},"confidence":"high","strength":"normal"}],"depth":3,"sources":[{"title":"Jacobs, Jordan, Nowlan & Hinton (1991), Adaptive Mixtures of Local Experts","tier":"reference"},{"title":"Shazeer et al. (2017), Outrageously Large Neural Networks - The Sparsely-Gated Mixture-of-Experts Layer","tier":"reference"},{"title":"Fedus, Zoph & Shazeer (2022), Switch Transformers","tier":"reference"}],"draft":true}