#sparse-models

EMO: How Constraining Expert Selection to Document-Level Pools Causes Semantic Modularity to Emerge During MoE Pretraining