Mixture of experts (MoE)

In a sparse MoE layer a router, usually a single linear projection W_g, scores each token's hidden state x against N experts; the top k scores are kept and renormalised with a softmax, and the layer output is y = Σ g_i(x) · E_i(x) over the selected experts only. In transformer language models the experts are copies of the feed-forward (MLP) sublayer, while attention remains dense and shared. Routing is decided independently per token and per layer, so a single sentence passes through many different expert combinations. The original idea (Jacobs, Jordan, Nowlan and Hinton, 1991) used a gating network to blend a few whole models; Shazeer et al. (2017) made it sparse and conditional with noisy top-k gating between LSTM layers, reaching 137 billion parameters.

The main engineering problem is load balancing. Without a counter-pressure the router collapses onto a few experts that then get all the training signal. GShard (Lepikhin et al., 2020) used top-2 routing with an auxiliary balancing loss; Switch Transformer (Fedus, Zoph and Shazeer, 2022) simplified to top-1 routing, defined an expert capacity (tokens per expert per batch, scaled by a capacity factor) beyond which tokens are dropped and passed on via the residual path, and trained models up to 1.6 trillion parameters. ST-MoE added a router z-loss for numerical stability. Later designs use many small fine-grained experts plus one or more always-active shared experts, and DeepSeek-V3 (671 billion total parameters, about 37 billion activated per token) balances load with a per-expert bias term instead of a large auxiliary loss.

The accounting distinction is between total and active parameters. Mixtral 8x7B has 8 experts per layer with top-2 routing, 46.7 billion total parameters and 12.9 billion used per token; its compute per token resembles a 13-billion dense model, but at 16-bit precision all weights still need roughly 94 GB of accelerator memory. At small batch sizes decoding is memory-bandwidth-bound and only active experts are read, so MoE is fast; at large batch sizes different tokens hit different experts, most weights are read anyway, and the advantage shrinks. Across many devices, expert parallelism places experts on different GPUs and requires all-to-all communication of tokens in every MoE layer, which makes interconnect bandwidth a bottleneck.

A frequent misconception is that experts are topical specialists, one for law and one for medicine. Analyses such as the Mixtral paper found little domain-level specialisation; routing correlates more with token types and syntax. MoE models have also been reported to be harder to fine-tune stably and more prone to overfitting on small datasets than dense models of similar active size, and routing adds a source of non-determinism when capacity limits drop tokens depending on what else is in the batch.