Beyond Geometric Complementarity: Coherent Overlap in Sparse Mixture-of-Experts Routing

A new study finds that sparse mixture-of-experts language models often route tokens to experts whose representations overlap.

The researchers separated routing coherence, expert quality, and token-context effects, showing that geometric similarity alone does not prove redundancy or pruning potential.

Across OLMoE, Mixtral, and DeepSeek-style models, expert subspaces overlapped substantially, yet the routed experts explained token representations better than matched alternatives.

Later experts often improved next-token prediction, and in the controlled experiments Top-2 training outperformed Top-1.