论文信息
标题:Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
作者:William Fedus, Barret Zoph, Noam Shazeer, (Google 2021)
摘要
MoE (Mixture of Expert) 模型的特点:
- select different parameters for each incoming example.
- a sparsely-activated model – with an outrageous number of parameters – but a constant computational cost.
- 限制MoE广泛使用的因素&#