Mixture of Experts: How Sparse Gating Enables 1T-Parameter LLMs

A 1-trillion-parameter model sounds impossible to run — until you realize that most of those parameters are inactive for any given token… Continue reading on Towards AI »