Mix-MoE: Improving Multilingual Machine Translation of Large Language Models through Mixed MoEs

Researchers introduced Mix-MoE, a two-stage Mixture-of-Experts framework for multilingual machine translation in LLMs.

The model first learns from monolingual data, then from parallel data, with separate expert groups for language modeling and translation.

A Fourier-based routing mechanism helps direct inputs to the right experts and reduces task interference.

Experiments showed Mix-MoE outperformed baseline methods and better preserved monolingual knowledge while improving translation quality.