Sparse MoE (Outrageously Large Neural Networks・Shazeer 2017年-2024年)（スパースモエ）
2017年Shazeer + Google発表Sparse MoE paper・Industry-leading Sparse Mixture of Experts paradigm Pioneer + Industry-leading Outrageously Large Neural Networks + Industry-leading sparsely-gated MoE + Industry-leading MoE Pioneer foundation。
概要
Sparse MoE は、2017年Shazeer et al. + Google発表"Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" paper・Industry-leading Sparse Mixture of Experts paradigm Pioneer 2017年-2024年 + Industry-leading MoE Pioneer foundation position確立。Sparse MoE specifications = Industry-leading Sparse Mixture of Experts paradigm Pioneer (Industry-leading Sparse MoE + Industry-leading Sparse Mixture of Experts paradigm + Industry-leading MoE Pioneer foundation) + Industry-leading Outrageously Large Neural Networks (Industry-leading Outrageously Large Neural Networks paper + Industry-leading sparsely-gated MoE Layer + Industry-leading Shazeer + Google paper) + Industry-leading sparsely-gated MoE (Industry-leading sparsely-gated Mixture-of-Experts Layer + Industry-leading top-k gating + Industry-leading sparse expert activation) + Industry-leading MoE Pioneer foundation (Industry-leading MoE Pioneer foundation + Industry-leading Switch Transformer/GLaM/Mixtral successors Pioneer)。
主な特徴・仕組み
- Authoring: Shazeer et al. + Google 2017年発表
- Paper: Industry-leading Outrageously Large Neural Networks
- Paradigm: Industry-leading Sparse Mixture of Experts Pioneer
- Mechanism: Industry-leading sparsely-gated MoE Layer + top-k gating
- Heritage: Industry-leading 7年heritage MoE Pioneer foundation
- Industry Position: Industry-leading Sparse MoE paradigm Pioneer
- Shazeer: Industry-leading Noam Shazeer Transformer co-inventor + MoE Pioneer
- Successors: Industry-leading Switch Transformer + GLaM + Mixtral + Expert Choice
- Industry-Leading: Industry-leading Sparse MoE paradigm Pioneer foundation
- Use Cases: Industry-leading MoE LLM training paradigm foundation
スペック比較表
| LLM Mixture of Experts (2017-2024) | Approach | Heritage | Industry Position |
|---|---|---|---|
| Sparse MoE (Shazeer) | Sparsely-gated MoE Layer Pioneer | 2017年Google Pioneer | Industry-leading Sparse MoE Pioneer foundation |
| Switch Transformer (Google) | Single expert per token | 2021年Google | Industry-leading single-expert switch |
| GLaM (Google) | Generalist Language Model MoE | 2021年Google | Industry-leading GLaM Google MoE |
| Mixtral 8x7B (Mistral) | 8x7B MoE open-source | 2023年Mistral AI | Industry-leading open-source MoE flagship |
| Expert Choice Routing | Expert-side routing | 2022年Google | Industry-leading Expert Choice routing |
具体例・対応製品
- Sparse MoE (Shazeer + Google・2017年-2024年): Industry-leading Sparse MoE paradigm Pioneer
- Industry-leading Outrageously Large Neural Networks paper: Industry-leading 2017年paper
- Industry-leading sparsely-gated MoE Layer: Industry-leading sparsely-gated MoE
- Industry-leading Noam Shazeer Transformer co-inventor + MoE Pioneer: Industry-leading Shazeer multi-Pioneer
- Industry-leading Switch Transformer + GLaM + Mixtral + Expert Choice successors: Industry-leading successors
- 競合 Switch Transformer (Google 2021年) + Mixtral 8x7B (Mistral 2023年): Industry-leading MoE successor competitors
自作PCでの選び方・注意点
Sparse MoE は「Industry-leading Sparse Mixture of Experts paradigm Pioneer + Industry-leading Outrageously Large Neural Networks」「Industry-leading sparsely-gated MoE + Industry-leading MoE Pioneer foundation」用途のIndustry-leading Sparse MoE Pioneer 2017年Shazeer + Google product。Industry-leading sparsely-gated MoE Layer (Industry-leading sparsely-gated Mixture-of-Experts Layer + Industry-leading top-k gating + Industry-leading sparse expert activation + Industry-leading Outrageously Large Neural Networks paper) で Industry-leading sparsely-gated MoE Layer + Industry-leading top-k gating + Industry-leading sparse expert activation。Industry-leading MoE Pioneer foundation (Industry-leading MoE Pioneer foundation + Industry-leading Switch Transformer/GLaM/Mixtral successors Pioneer + Industry-leading 7年heritage MoE Pioneer foundation) で Industry-leading MoE Pioneer foundation + Industry-leading 7年heritage + Industry-leading successors Pioneer。Industry-leading Shazeer multi-Pioneer (Industry-leading Noam Shazeer Transformer co-inventor + MoE Pioneer + Industry-leading Shazeer GLU Variants + AdamW + multi-Pioneer Google) で Industry-leading Noam Shazeer Transformer co-inventor + Industry-leading multi-Pioneer Google AI backing。但しIndustry-leading Switch Transformer/GLaM/Mixtral/Expert Choice 2021-2023 successors competition (Industry-leading Switch Transformer Google 2021年single-expert + GLaM Google 2021年 + Mixtral Mistral 2023年8x7B + Expert Choice Google 2022年 vs Sparse MoE Shazeer 2017年Pioneer trade-off) で Industry-leading successors 4-paradigm evolution vs Sparse MoE Pioneer + Industry-leading 7年Pioneer foundation adoption alignment必須。
関連用語との違い
- vs Switch Transformer (Google 2021年): Sparse MoEはSparsely-gated Pioneer + top-k 2017年・Switch Transformerは Single expert per token + Google 2021年
- vs GLaM (Google 2021年): Sparse MoEは Pioneer foundation + 2017年・GLaMはGeneralist Language Model MoE + Google 2021年
- vs Mixtral 8x7B (Mistral 2023年): Sparse MoEは Pioneer paradigm + 2017年・Mixtralは8x7B open-source + Mistral 2023年commercial
よくある質問（FAQ）
Q1: Sparse MoE vs Switch Transformer 違いは？ A: Sparse MoE (Shazeer + Google 2017年 + sparsely-gated MoE Layer + top-k gating + MoE Pioneer foundation + Outrageously Large Neural Networks paper) vs Switch Transformer (Google 2021年 + Single expert per token + simplified routing + Switch Transformer Google)・Industry-leading sparsely-gated + top-k + Pioneer foundation = Sparse MoE + Industry-leading Single expert per token + Switch simplified = Switch Transformer preference judgment。
Q2: Industry-leading sparsely-gated MoE Layer value は？ A: Industry-leading sparsely-gated (Industry-leading sparsely-gated Mixture-of-Experts Layer + Industry-leading top-k gating + Industry-leading sparse expert activation + Industry-leading Shazeer 2017年Pioneer paradigm)。
Q3: Industry-leading MoE Pioneer foundation value は？ A: Industry-leading MoE Pioneer (Industry-leading MoE Pioneer foundation + Industry-leading Switch Transformer/GLaM/Mixtral successors Pioneer + Industry-leading 7年heritage MoE Pioneer foundation + Industry-leading Sparse MoE paradigm foundation)。
まとめ
Sparse MoE = 2017年Shazeer et al. + Google発表のOutrageously Large Neural Networks paper Sparse Mixture of Experts paradigm Pioneer。Industry-leading sparsely-gated Mixture-of-Experts Layer + Industry-leading top-k gating + Industry-leading sparse expert activation + Industry-leading Noam Shazeer Transformer co-inventor + MoE Pioneer + Industry-leading Switch Transformer + GLaM + Mixtral + Expert Choice Industry-leading successors Pioneer foundation + Industry-leading 7年heritage MoE Pioneer foundation position確立。