Mixture of Experts (MoE) architectures let a model have a huge total parameter count while only activating a fraction of those parameters for any given input — a genuinely clever way to scale capability without proportionally scaling inference cost.
The Core Idea
Instead of one single, dense feed-forward network per layer, an MoE layer has MANY smaller "expert" networks. A lightweight "router" network looks at each token and decides which small subset of experts (e.g. 2 out of 64) should actually process it - the rest of the experts are simply skipped for that token.
Why This Is Genuinely Efficient
A dense model with the same total parameter count would need to run every single parameter for every single token — genuinely expensive. An MoE model with, say, 8 experts activated per token out of 64 total experts uses only a fraction of its total parameters for any given inference, while still having access to the full breadth of specialized knowledge distributed across all 64 experts collectively over the course of many different inputs.
The Real Trade-off
MoE models are genuinely more complex to train (routing decisions add real instability risk during training) and need to keep ALL experts loaded in memory even though only a few are active per token — meaning MoE saves on compute per token, but doesn’t necessarily save on total memory footprint the way a smaller dense model would. This is exactly why MoE tends to appear specifically in very large-scale models chasing maximum capability per unit of inference compute, rather than in models optimized purely for minimal memory footprint.