Mixtral 8x7B (Jiang 2023年)（ミクストラル8x7B）
2023年Mistral AI (Jiang et al.)発表Mixtral 8x7B・Industry-leading first open-source MoE LLM + Industry-leading 8 experts top-2 routing + Industry-leading 47B params/13B active + Industry-leading Mistral AI flagship MoE。
概要
Mixtral 8x7B は、2023年Mistral AI (Jiang et al.)発表Mixtral 8x7B・Industry-leading first open-source MoE LLM 2023年 + Industry-leading Mistral AI flagship MoE position確立。Mixtral 8x7B specifications = Industry-leading first open-source MoE LLM (Industry-leading Mixtral 8x7B Jiang 2023 + Industry-leading first open-source MoE + Industry-leading Mistral AI flagship) + Industry-leading 8 experts top-2 routing + Industry-leading 47B params/13B active + Industry-leading Mistral AI flagship MoE。
主な特徴・仕組み
- Authors: Industry-leading Albert Q. Jiang + Alexandre Sablayrolles + Antoine Roux + Arthur Mensch + Blanche Savary + Chris Bamford + Devendra Singh Chaplot + Diego de las Casas + Emma Bou Hanna + Florian Bressand + Gianna Lengyel + Guillaume Bour + Guillaume Lample + Lélio Renard Lavaud + Lucile Saulnier + Marie-Anne Lachaux + Pierre Stock + Sandeep Subramanian + Sophia Yang + Szymon Antoniak + Teven Le Scao + Thibaut Lavril + Thomas Wang + Timothée Lacroix + William El Sayed (Mistral AI)
- Year: 2023年 (arXiv 2024年1月発表 + Apache 2.0 release 2023年12月)
- Paper: Industry-leading "Mixtral of Experts" arXiv 2401.04088
- Method Type: Industry-leading Sparse Mixture of Experts MoE LLM
- MoE Config: Industry-leading 8 experts top-2 routing per token
- Total Params: Industry-leading 46.7B total parameters
- Active Params: Industry-leading 12.9B active parameters per token
- First Open MoE: Industry-leading first open-source production MoE LLM
- License: Industry-leading Apache 2.0 open license
- Performance: Industry-leading Mixtral 8x7B > LLaMA 2 70B + GPT-3.5
- Industry-Leading: Industry-leading first open-source MoE LLM Mistral AI 2023
スペック比較表
| LLM MoE (2023-2024) | Year | Org | Config | Total/Active | Industry Position |
|---|---|---|---|---|---|
| Mixtral 8x7B | 2023 | Mistral AI | 8 experts top-2 | 47B/13B | Industry-leading first open MoE |
| Mixtral 8x22B Instruct | 2024 | Mistral AI | 8 experts top-2 | 141B/39B | Industry-leading Mistral large MoE |
| DBRX MoE | 2024 | Databricks | 16 experts top-4 | 132B/36B | Industry-leading Databricks MoE |
| Qwen1.5-MoE-A2.7B | 2024 | Alibaba Qwen | 60 experts top-4 | 14.3B/2.7B | Industry-leading Qwen small MoE |
| Grok-1 MoE | 2024 | xAI | 8 experts top-2 | 314B/86B | Industry-leading xAI Grok MoE |
具体例・対応製品
- Mixtral 8x7B (2023年, Mistral AI Jiang et al.): Industry-leading first open-source MoE LLM
- Industry-leading 8 experts top-2 routing per token: Industry-leading 8 experts top-2
- Industry-leading 46.7B total + 12.9B active parameters: Industry-leading 47B/13B active
- Industry-leading Apache 2.0 open license: Industry-leading Apache 2.0
- Industry-leading Mixtral 8x7B > LLaMA 2 70B + GPT-3.5 performance: Industry-leading > LLaMA 2 70B + GPT-3.5
- 競合 Mixtral 8x22B Instruct + DBRX MoE + Qwen1.5-MoE + Grok-1 MoE: Industry-leading MoE competitors
選び方・注意点
Mixtral 8x7B は「Industry-leading first open-source MoE LLM + 8 experts top-2 routing」「Industry-leading 47B/13B active + Mistral AI flagship」用途のIndustry-leading 2023年Mistral AI Jiang et al.発表product。Industry-leading 8 experts top-2 (Industry-leading 8 experts top-2 routing per token + Industry-leading Mixtral 8 experts signature + Industry-leading top-2 routing efficiency) で Industry-leading 8 experts top-2 + Industry-leading Mixtral signature + Industry-leading top-2 routing efficiency。Industry-leading 47B/13B active (Industry-leading 46.7B total + 12.9B active parameters + Industry-leading Mixtral 47B/13B signature + Industry-leading sparse activation efficiency) で Industry-leading 47B/13B + Industry-leading Mixtral signature + Industry-leading sparse activation。Industry-leading first open MoE (Industry-leading first open-source production MoE LLM + Industry-leading Mistral AI first MoE + Industry-leading Apache 2.0 release December 2023) で Industry-leading first open MoE + Industry-leading Mistral first + Industry-leading Apache 2.0 release。Industry-leading performance (Industry-leading Mixtral 8x7B > LLaMA 2 70B + GPT-3.5 performance + Industry-leading Mistral AI flagship performance + Industry-leading benchmark superiority) で Industry-leading > LLaMA 2 70B + GPT-3.5 + Industry-leading Mistral flagship + Industry-leading benchmark superiority。Industry-leading Mistral AI (Industry-leading Mistral AI flagship MoE + Industry-leading Arthur Mensch + Guillaume Lample + Albert Jiang Mistral AI founders ex-DeepMind+Meta + Industry-leading Paris France 2023 startup) で Industry-leading Mistral AI flagship + Industry-leading Mensch+Lample+Jiang + Industry-leading Paris startup 2023。但しIndustry-leading Mixtral 8x22B Instruct + DBRX MoE + Qwen1.5-MoE + Grok-1 MoE competition で Industry-leading 4-MoE competitors vs Mixtral 8x7B + Industry-leading Sparse Mixture of Experts MoE LLM + 8 experts top-2 routing + 46.7B total + 12.9B active + Apache 2.0 open license + > LLaMA 2 70B + GPT-3.5 + first open-source production MoE LLM + Albert Q. Jiang + Arthur Mensch + Guillaume Lample Mistral AI Paris France 2023 unique advantage adoption alignment必須。
関連用語との違い
- vs Mixtral 8x22B Instruct (2024): Mixtral 8x7Bは47B/13B + first open MoE・8x22Bは141B/39B + Mistral large
- vs DBRX MoE (Databricks 2024): Mixtral 8x7Bは8 experts top-2 Mistral・DBRXは16 experts top-4 Databricks
- vs Qwen1.5-MoE-A2.7B (Alibaba 2024): Mixtral 8x7Bは8 experts large 47B・Qwen1.5-MoEは60 experts small 14.3B
よくある質問（FAQ）
Q1: Mixtral 8x7B vs DBRX MoE 違いは？ A: Mixtral 8x7B (Industry-leading 8 experts top-2 routing + 46.7B total + 12.9B active + Apache 2.0 + first open-source MoE + > LLaMA 2 70B + GPT-3.5 + Albert Jiang + Arthur Mensch + Guillaume Lample Mistral AI Paris 2023) vs DBRX MoE (Industry-leading 16 experts top-4 routing + 132B total + 36B active + Databricks open + DBRX MoE 2024)・Industry-leading 8 experts top-2 + 47B/13B + Apache 2.0 + Mistral + Paris + 2023 = Mixtral 8x7B + Industry-leading 16 experts top-4 + 132B/36B + Databricks + 2024 = DBRX preference judgment。
Q2: Industry-leading 8 experts top-2 + first open MoE value は？ A: Industry-leading 8 experts + first open (Industry-leading 8 experts top-2 routing per token + Industry-leading first open-source production MoE LLM + Industry-leading Mixtral dual signature)。
Q3: Industry-leading 47B/13B active + Mistral AI flagship value は？ A: Industry-leading 47B/13B + Mistral (Industry-leading 46.7B total + 12.9B active parameters + Industry-leading Mistral AI flagship MoE + Industry-leading Arthur Mensch + Guillaume Lample + Albert Jiang Mistral Paris 2023)。
まとめ
Mixtral 8x7B = 2023年Mistral AI (Jiang et al.)発表のfirst open-source MoE LLM。Industry-leading Sparse Mixture of Experts MoE LLM + Industry-leading 8 experts top-2 routing per token + Industry-leading 46.7B total parameters + Industry-leading 12.9B active parameters per token + Industry-leading Apache 2.0 open license + Industry-leading Mixtral 8x7B > LLaMA 2 70B + GPT-3.5 performance + Industry-leading Albert Q. Jiang + Arthur Mensch + Guillaume Lample + Alexandre Sablayrolles Mistral AI authors + Industry-leading "Mixtral of Experts" arXiv 2401.04088 + Industry-leading Mistral AI Paris France 2023 startup + Industry-leading first open-source MoE LLM Mistral AI 2023 position確立。