EXPLAINER

Mixture of Experts (MoE)

techniquefreshness: evolvingsources not yet verified

Mixture-of-experts models contain many specialist sub-networks and route each token to only a few, giving large capacity at lower per-token compute.

Short answer: A router sends each token to a small subset of expert sub-networks, so total parameters are large but active parameters per token are fewer. Memory still holds all experts; routing and load balancing add complexity.

What is it?

Instead of every part of the model working on every word, an MoE model has many "expert" blocks and a router that picks a few for each word. The model can be very big in total yet use only part of it each time.

Also known as: MoE, sparse mixture of experts, expert routing, sparse models, Switch Transformer, Mixtral, top-k routing, active parameters.

Technical definition

In sparse MoE transformer layers, the feed-forward block is replaced by N experts and a gating network selecting top-k experts per token (Shazeer et al., 2017; Switch Transformer, Fedus et al., 2021; Mixtral, Jiang et al., 2024). Auxiliary losses encourage load balance. Compute per token scales with active experts, but memory scales with total parameters and expert-parallel serving needs all-to-all communication. Total and active parameter counts differ and should not be conflated.

Entity type: technique · Freshness class: evolving.

When should I use it?

Relevant when choosing or serving large open models; rarely something application teams build.

Implementation patterns

  • Top-1 or top-2 routing with load-balancing loss.
  • Expert parallelism across GPUs.
  • Shared experts alongside routed ones.
  • Quantise experts to cut memory.

Technologies and frameworks

Commonly associated: Switch Transformer, Mixtral, DeepSpeed-MoE, Megatron-LM, vLLM. Names are examples of what exists, not recommendations, and projects change quickly.

Practical guide

  1. Compare both total and active parameters.
  2. Estimate memory from total, compute from active.
  3. Choose a serving engine with MoE support.
  4. Benchmark latency and throughput on your hardware.

Common failure modes

  • Load imbalance, with some experts overused.
  • High memory despite low active compute.
  • Communication bottlenecks at scale.
  • Fine-tuning instability.

Troubleshooting

  • Cannot fit the model in memory — Quantise, offload, or use fewer experts/smaller variant.
  • Throughput lower than expected — Check batching, expert parallelism and communication overhead.

Contrasted with

Questions this page answers

  • What is a mixture of experts model?
  • What are active parameters in an MoE?
  • Why are MoE models cheaper to run per token?
  • Do MoE models need less memory?

Typical goals: understand why some models quote total and active parameters; estimate serving cost of MoE models.

Sources and verification

Status: not independently verified. This entry was authored on 2026-10-06 from general technical knowledge, without live checking of the sources below. Treat every source as a lead to confirm (identifier, title, authorship, current version) before publication or citation.

Freshness class: evolving — Established techniques whose best practice shifts as research and tooling move. Re-review every 6 months.

Canonical sources (unverified leads):

  • Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017) — arXiv:1701.06538
  • Fedus et al., Switch Transformers (2021) — arXiv:2101.03961
  • Jiang et al., Mixtral of Experts (2024) — arXiv:2401.04088