Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions across depth can often be predicted from earlier routing signals, suggesting that routing is not fully independent acro...