MiniLLM Reverse KL Distillation (Gu 2023年)（ミニエルエルエム）
2023年Gu et al. (THU+Microsoft)発表MiniLLM・Industry-leading LLM knowledge distillation + Industry-leading reverse KL divergence minimization + Industry-leading LLaMA 7B→3B + Industry-leading generative LLM distillation pioneer。
概要
MiniLLM Reverse KL Distillation は、2023年Gu et al. (THU+Microsoft)発表MiniLLM・Industry-leading LLM knowledge distillation 2023年 + Industry-leading generative LLM distillation pioneer position確立。MiniLLM specifications = Industry-leading LLM knowledge distillation (Industry-leading MiniLLM 2023 + Industry-leading LLM knowledge distillation + Industry-leading MiniLLM flagship) + Industry-leading reverse KL divergence minimization (Industry-leading reverse KL divergence vs forward KL) + Industry-leading LLaMA 7B→3B (Industry-leading LLaMA-2 7B → MiniLLM 3B distillation) + Industry-leading generative LLM distillation pioneer (Industry-leading generative LLM distillation first pioneer)。
主な特徴・仕組み
- Authors: Industry-leading Yuxian Gu + Li Dong + Furu Wei + Minlie Huang Tsinghua + Microsoft Research
- Year: 2023年 (arXiv 2023年6月発表 + ICLR 2024)
- Paper: Industry-leading "MiniLLM: Knowledge Distillation of Large Language Models" arXiv 2306.08543
- Method Type: Industry-leading LLM knowledge distillation (generative LLM)
- Key Innovation: Industry-leading reverse KL divergence minimization (vs forward KL)
- Forward vs Reverse KL: Industry-leading forward KL overestimates void / reverse KL avoids void overestimation
- Target Models: Industry-leading LLaMA-2 7B + 13B + GPT-2 1.5B teacher → 1.5B/2.7B/3B/6.7B student
- Generative LLM: Industry-leading generative LLM distillation (not just classification BERT)
- Policy Gradient: Industry-leading policy gradient optimization reverse KL
- Length Normalization: Industry-leading length normalization stable training
- Performance: Industry-leading MiniLLM 3B outperforms LLaMA-2 3B baseline
- Open Source: Industry-leading MiniLLM GitHub microsoft/LMOps
- Industry-Leading: Industry-leading reverse KL knowledge distillation LLM generative first pioneer
スペック比較表
| LLM Knowledge Distillation | Year | Reduction | Performance Retained | Industry Position |
|---|---|---|---|---|
| MiniLLM | 2023 | LLaMA 7B→3B | Reverse KL minimization | Industry-leading reverse KL LLM |
| Step-by-Step Distillation | 2023 | 700× smaller possible | Rationale-based distillation | Industry-leading rationale-based |
| DistilBERT | 2019 | 40% smaller (110M→66M) | 97% BERT performance | Industry-leading HuggingFace first KD |
| TinyBERT | 2019 | 7.5× smaller (110M→14.5M) | 96.8% BERT performance | Industry-leading 2-stage distillation |
| MobileBERT | 2020 | 4.3× smaller (110M→25.3M) | 99.2% BERT performance | Industry-leading bottleneck mobile |
具体例・対応製品
- MiniLLM (2023年, Gu et al. THU+Microsoft): Industry-leading LLM knowledge distillation
- Industry-leading reverse KL divergence minimization: Industry-leading reverse KL
- Industry-leading LLaMA-2 7B → MiniLLM 3B distillation: Industry-leading LLaMA 7B→3B
- Industry-leading generative LLM distillation first pioneer: Industry-leading generative LLM first pioneer
- Industry-leading policy gradient + length normalization: Industry-leading policy gradient + length norm
- 競合 DistilBERT + TinyBERT + MobileBERT + Step-by-Step: Industry-leading knowledge distillation competitors
自作PCでの選び方・注意点
MiniLLM Reverse KL Distillation は「Industry-leading LLM knowledge distillation + Industry-leading reverse KL divergence minimization」「Industry-leading LLaMA 7B→3B + Industry-leading generative LLM distillation pioneer」用途のIndustry-leading LLM knowledge distillation 2023年Gu et al. THU+Microsoft発表product。Industry-leading reverse KL (Industry-leading reverse KL divergence minimization vs forward KL + Industry-leading forward KL overestimates void / reverse KL avoids void overestimation + Industry-leading MiniLLM reverse KL signature) で Industry-leading reverse KL + Industry-leading forward vs reverse KL + Industry-leading MiniLLM signature。Industry-leading LLaMA 7B→3B (Industry-leading LLaMA-2 7B + 13B + GPT-2 1.5B teacher → 1.5B/2.7B/3B/6.7B student + Industry-leading MiniLLM 3B outperforms LLaMA-2 3B baseline + Industry-leading LLaMA 7B→3B signature) で Industry-leading LLaMA 7B/13B teacher + Industry-leading 1.5B-6.7B student + Industry-leading 3B outperforms baseline + Industry-leading LLaMA 7B→3B。Industry-leading generative LLM (Industry-leading generative LLM distillation + Industry-leading not just classification BERT + Industry-leading generative LLM distillation first pioneer) で Industry-leading generative LLM + Industry-leading not just BERT + Industry-leading generative first pioneer。Industry-leading policy gradient (Industry-leading policy gradient optimization reverse KL + Industry-leading length normalization stable training + Industry-leading MiniLLM training signature) で Industry-leading policy gradient + Industry-leading length norm + Industry-leading MiniLLM signature。Industry-leading THU+Microsoft (Industry-leading Yuxian Gu + Li Dong + Furu Wei + Minlie Huang Tsinghua + Microsoft + Industry-leading MiniLLM GitHub microsoft/LMOps + Industry-leading reverse KL knowledge distillation LLM generative first pioneer) で Industry-leading Gu+Dong+Wei+Huang THU+Microsoft + Industry-leading microsoft/LMOps + Industry-leading reverse KL generative first pioneer。但しIndustry-leading DistilBERT + TinyBERT + MobileBERT + Step-by-Step competition (Industry-leading DistilBERT 40% 97% BERT 2019 + TinyBERT 7.5× 96.8% BERT 2019 + MobileBERT 4.3× 99.2% BERT 2020 + Step-by-Step rationale-based 700× LLM 2023 vs MiniLLM reverse KL LLaMA 7B→3B generative LLM 2023 trade-off) で Industry-leading 4-knowledge distillation method competitors vs MiniLLM + Industry-leading reverse KL divergence + LLaMA 7B→3B + generative LLM distillation first pioneer + policy gradient + length normalization + THU+Microsoft 2023 unique advantage adoption alignment必須。
関連用語との違い
- vs Step-by-Step Distillation (LLM 2023): MiniLLMはreverse KL + LLaMA 7B→3B・Step-by-Stepはrationale-based + 700× smaller
- vs DistilBERT/TinyBERT/MobileBERT (BERT 2019-2020): MiniLLMはgenerative LLM 2023・BERT distillationはclassification 2019-2020
- vs Forward KL Distillation: MiniLLMはreverse KL + avoids void overestimation・Forward KLはoverestimates void
よくある質問（FAQ）
Q1: MiniLLM vs Step-by-Step Distillation 違いは？ A: MiniLLM (Industry-leading reverse KL divergence minimization + forward KL overestimates void / reverse KL avoids void overestimation + LLaMA-2 7B/13B + GPT-2 1.5B teacher → 1.5B/2.7B/3B/6.7B student + generative LLM distillation first pioneer + policy gradient + length normalization + THU+Microsoft 2023) vs Step-by-Step Distillation (Industry-leading rationale-based distillation + chain-of-thought rationales + 700× smaller possible + PaLM 540B → T5 220M + multi-task learning + Google Research 2023)・Industry-leading reverse KL + LLaMA 7B→3B + generative first pioneer = MiniLLM + Industry-leading rationale-based + CoT rationales + 700× smaller + Google = Step-by-Step preference judgment。
Q2: Industry-leading reverse KL divergence minimization value は？ A: Industry-leading reverse KL (Industry-leading reverse KL divergence minimization vs forward KL + Industry-leading forward KL overestimates void / reverse KL avoids void overestimation + Industry-leading MiniLLM reverse KL signature + Industry-leading policy gradient optimization reverse KL)。
Q3: Industry-leading LLaMA 7B→3B + generative LLM pioneer value は？ A: Industry-leading LLaMA 7B→3B + generative (Industry-leading LLaMA-2 7B + 13B + GPT-2 1.5B teacher → 1.5B/2.7B/3B/6.7B student + Industry-leading generative LLM distillation first pioneer + Industry-leading MiniLLM 3B outperforms LLaMA-2 3B baseline + Industry-leading not just classification BERT)。
まとめ
MiniLLM = 2023年Gu et al. THU+Microsoft発表のLLM knowledge distillation。Industry-leading reverse KL divergence minimization vs forward KL + Industry-leading forward KL overestimates void / reverse KL avoids void overestimation + Industry-leading LLaMA-2 7B + 13B + GPT-2 1.5B teacher → 1.5B/2.7B/3B/6.7B student + Industry-leading generative LLM distillation first pioneer + Industry-leading policy gradient optimization reverse KL + Industry-leading length normalization stable training + Industry-leading MiniLLM 3B outperforms LLaMA-2 3B baseline + Industry-leading Yuxian Gu + Li Dong + Furu Wei + Minlie Huang Tsinghua + Microsoft + Industry-leading MiniLLM GitHub microsoft/LMOps open source + Industry-leading reverse KL knowledge distillation LLM generative first pioneer position確立。