MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
Mixture-of-Experts (MoE) models scale large language models efficiently by sparsely activating experts, but once an expert is selected, it is executed fully. Hence, the trade-off between accuracy and computation in an MoE model typically exhibits large discontinuities. We propose Mixture of Slimmable Experts (MoSE), an MoE architecture in which each expert has a nested, slimmable structure that can be executed at variable widths. This enables conditional computation not only over which experts are activated but also over how much of each expert is utilized. Consequently, a single pretrained MoSE model can support a more continuous spectrum of accuracy-compute trade-offs at inference time. We present a simple and stable training recipe for slimmable experts under sparse routing, combining multi-width training with standard MoE objectives. During inference, we explore strategies for runtime width determination, including a lightweight test-time training mechanism that learns how to map router confidence/probabilities to expert widths under a fixed budget. Experiments on GPT-style models, various routing regimes, zero-shot downstream reasoning benchmarks, and continual pre-training adaptation of DeepSeek model show that MoSE matches or improves standard MoE at full width and consistently shifts the compute-quality frontier toward lower inference FLOPs.
Authors: Nurbek Tastan, Stefanos Laskaridis, Karthik Nandakumar, Samuel Horvàth
Published at: International Conference on Machine Learning (ICML’26)
Overview
Standard Mixture-of-Experts models only give you conditional computation over which experts run, not how much of each expert runs, which means the accuracy-compute trade-off jumps in large, discontinuous steps. MoSE closes this gap by giving each expert a nested, slimmable structure that can be executed at variable widths, so a single pretrained model supports a much more continuous spectrum of accuracy-compute trade-offs.
We introduce a simple, stable recipe for training slimmable experts under sparse routing, combining multi-width training with standard MoE objectives, and explore runtime strategies for picking expert widths, including a lightweight test-time training mechanism that maps router confidence to widths under a fixed compute budget. Across GPT-style models, multiple routing regimes, zero-shot reasoning benchmarks, and continual pre-training of DeepSeek, MoSE matches or improves standard MoE at full width while consistently shifting the compute-quality frontier toward lower inference FLOPs.
Links
Reference
@inproceedings{
tastan2026mose,
title={MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models},
author={Tastan, Nurbek and Laskaridis, Stefanos and Nandakumar, Karthik and Horv{\`a}th, Samuel},
booktitle={International Conference on Machine Learning},
year={2026}
}
