Mixture-of-Experts (MoE)

I’m lacking good mental model for how this works.

I think the stanford CS336 lecture on MoE is probably good.

Some interesting ideas that I need to wrap my head around:

• Fine-grained experts vs coarse experts
  • if you’re restricted to the same number of parameters, perhaps it is better to further split them up, rather than have a larger parameter FFN
    • because you are adding those numbers at the end, so maybe there’s a way to represent richer information?
  • Relation to more # of shared experts?

Resources

• https://huggingface.co/blog/moe

Citation

(Shazeer et al., 2017)

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., & Dean, J. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. https://arxiv.org/abs/1701.06538