Step-by-Step Distillation Rationale-Based (Hsieh 2023年)（ステップバイステップディスティレーション）
2023年Hsieh et al. (Google Research)発表Step-by-Step Distillation・Industry-leading rationale-based LLM knowledge distillation + Industry-leading chain-of-thought rationales + Industry-leading 700× smaller possible + Industry-leading PaLM 540B → T5 220M。
概要
Step-by-Step Distillation Rationale-Based は、2023年Hsieh et al. (Google Research)発表Step-by-Step Distillation・Industry-leading rationale-based LLM knowledge distillation 2023年 + Industry-leading 700× smaller possible position確立。Step-by-Step Distillation specifications = Industry-leading rationale-based LLM knowledge distillation (Industry-leading Step-by-Step Distillation 2023 + Industry-leading rationale-based LLM knowledge distillation + Industry-leading Step-by-Step flagship) + Industry-leading chain-of-thought rationales (Industry-leading CoT rationales as additional supervision) + Industry-leading 700× smaller possible (Industry-leading 700× smaller than teacher possible) + Industry-leading PaLM 540B → T5 220M (Industry-leading PaLM 540B teacher → T5 220M student)。
主な特徴・仕組み
- Authors: Industry-leading Cheng-Yu Hsieh + Chun-Liang Li + Chih-Kuan Yeh + Hootan Nakhost + Yasuhisa Fujii + Alexander Ratner + Ranjay Krishna + Chen-Yu Lee + Tomas Pfister Google Cloud AI Research + Univ. of Washington
- Year: 2023年 (arXiv 2023年5月発表 + ACL Findings 2023)
- Paper: Industry-leading "Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Less Model Size" arXiv 2305.02301
- Method Type: Industry-leading rationale-based LLM knowledge distillation
- Key Innovation: Industry-leading chain-of-thought (CoT) rationales as additional supervision
- Rationale Generation: Industry-leading teacher LLM generates rationales (not just labels)
- Multi-Task Learning: Industry-leading multi-task learning (label prediction + rationale prediction)
- Size Reduction: Industry-leading 700× smaller than teacher possible (540B → 770M typical)
- Target Models: Industry-leading PaLM 540B teacher → T5 220M/770M/3B student
- Data Efficiency: Industry-leading outperforms few-shot LLM with 12.5% of training data
- Performance: Industry-leading 770M student outperforms 540B PaLM teacher on 4/4 benchmarks
- Open Source: Industry-leading Step-by-Step Distillation GitHub google-research/distilling-step-by-step
- Industry-Leading: Industry-leading rationale-based LLM distillation Google Research
スペック比較表
| LLM Knowledge Distillation | Year | Reduction | Performance Retained | Industry Position |
|---|---|---|---|---|
| Step-by-Step Distillation | 2023 | 700× smaller possible (540B→770M) | 770M outperforms PaLM 540B | Industry-leading rationale-based |
| MiniLLM | 2023 | LLaMA 7B→3B | Reverse KL minimization | Industry-leading reverse KL LLM |
| DistilBERT | 2019 | 40% smaller (110M→66M) | 97% BERT performance | Industry-leading HuggingFace first KD |
| TinyBERT | 2019 | 7.5× smaller (110M→14.5M) | 96.8% BERT performance | Industry-leading 2-stage distillation |
| MobileBERT | 2020 | 4.3× smaller (110M→25.3M) | 99.2% BERT performance | Industry-leading bottleneck mobile |
具体例・対応製品
- Step-by-Step Distillation (2023年, Hsieh et al. Google Research): Industry-leading rationale-based LLM distillation
- Industry-leading chain-of-thought rationales as supervision: Industry-leading CoT rationales supervision
- Industry-leading 700× smaller than teacher possible: Industry-leading 700× smaller
- Industry-leading PaLM 540B → T5 220M/770M/3B: Industry-leading PaLM 540B → T5
- Industry-leading 770M outperforms 540B PaLM on 4/4 benchmarks: Industry-leading 770M outperforms 540B
- 競合 DistilBERT + TinyBERT + MobileBERT + MiniLLM: Industry-leading knowledge distillation competitors
自作PCでの選び方・注意点
Step-by-Step Distillation Rationale-Based は「Industry-leading rationale-based LLM knowledge distillation + Industry-leading chain-of-thought rationales」「Industry-leading 700× smaller possible + Industry-leading PaLM 540B → T5 220M」用途のIndustry-leading rationale-based LLM distillation 2023年Hsieh et al. Google Research発表product。Industry-leading CoT rationales (Industry-leading chain-of-thought rationales as additional supervision + Industry-leading teacher LLM generates rationales not just labels + Industry-leading Step-by-Step CoT signature) で Industry-leading CoT rationales + Industry-leading teacher generates rationales + Industry-leading Step-by-Step signature。Industry-leading multi-task (Industry-leading multi-task learning label prediction + rationale prediction + Industry-leading Step-by-Step multi-task signature + Industry-leading label+rationale supervision) で Industry-leading multi-task + Industry-leading Step-by-Step signature + Industry-leading label+rationale。Industry-leading 700× smaller (Industry-leading 700× smaller than teacher possible 540B → 770M typical + Industry-leading PaLM 540B teacher → T5 220M/770M/3B student + Industry-leading Step-by-Step 700× signature) で Industry-leading 700× smaller + Industry-leading PaLM 540B→T5 220M/770M/3B + Industry-leading Step-by-Step 700× signature。Industry-leading 770M outperforms 540B (Industry-leading 770M student outperforms 540B PaLM teacher on 4/4 benchmarks + Industry-leading 12.5% training data efficiency + Industry-leading student outperforms teacher signature) で Industry-leading 770M outperforms 540B + Industry-leading 12.5% data efficiency + Industry-leading student outperforms teacher。Industry-leading Google Research (Industry-leading Cheng-Yu Hsieh + Chun-Liang Li + Chih-Kuan Yeh Google Cloud AI Research + UW + Industry-leading Step-by-Step Distillation GitHub google-research/distilling-step-by-step + Industry-leading rationale-based LLM distillation Google Research) で Industry-leading Hsieh+Li+Yeh Google+UW + Industry-leading google-research/distilling-step-by-step + Industry-leading rationale-based Google Research。但しIndustry-leading DistilBERT + TinyBERT + MobileBERT + MiniLLM competition (Industry-leading DistilBERT 40% 97% 2019 + TinyBERT 7.5× 96.8% 2019 + MobileBERT 4.3× 99.2% 2020 + MiniLLM reverse KL LLaMA 7B→3B 2023 vs Step-by-Step 700× smaller PaLM 540B→T5 rationale-based 2023 trade-off) で Industry-leading 4-knowledge distillation method competitors vs Step-by-Step + Industry-leading rationale-based + CoT rationales + 700× smaller + PaLM 540B→T5 220M + 770M outperforms 540B + 12.5% data efficiency + Google Research 2023 unique advantage adoption alignment必須。
関連用語との違い
- vs MiniLLM (Reverse KL 2023): Step-by-Stepはrationale-based + CoT + 700× smaller・MiniLLMはreverse KL + LLaMA 7B→3B
- vs DistilBERT/TinyBERT/MobileBERT (BERT 2019-2020): Step-by-StepはLLM 2023 rationale-based・BERT distillationはclassification 2019-2020
- vs Forward KL Distillation: Step-by-Stepはrationale-based supervision・Forward KLはlabel-only supervision
よくある質問（FAQ）
Q1: Step-by-Step Distillation vs MiniLLM 違いは？ A: Step-by-Step Distillation (Industry-leading chain-of-thought rationales as additional supervision + teacher LLM generates rationales not just labels + multi-task learning label+rationale prediction + 700× smaller than teacher possible + PaLM 540B teacher → T5 220M/770M/3B student + 770M outperforms 540B PaLM on 4/4 benchmarks + 12.5% training data efficiency + Google Cloud AI Research+UW 2023) vs MiniLLM (Industry-leading reverse KL divergence minimization + forward KL overestimates void + LLaMA-2 7B/13B + GPT-2 1.5B teacher → 1.5B/2.7B/3B/6.7B student + generative LLM distillation first pioneer + policy gradient + length normalization + THU+Microsoft 2023)・Industry-leading rationale-based + CoT + 700× + 540B→T5 + 770M outperforms 540B = Step-by-Step + Industry-leading reverse KL + LLaMA 7B→3B + generative first pioneer = MiniLLM preference judgment。
Q2: Industry-leading CoT rationales + multi-task learning value は？ A: Industry-leading CoT + multi-task (Industry-leading chain-of-thought rationales as additional supervision + Industry-leading teacher LLM generates rationales not just labels + Industry-leading multi-task learning label prediction + rationale prediction + Industry-leading Step-by-Step multi-task signature)。
Q3: Industry-leading 700× smaller + 770M outperforms 540B value は？ A: Industry-leading 700× + 770M outperforms (Industry-leading 700× smaller than teacher possible 540B → 770M typical + Industry-leading PaLM 540B teacher → T5 220M/770M/3B student + Industry-leading 770M student outperforms 540B PaLM teacher on 4/4 benchmarks + Industry-leading 12.5% training data efficiency)。
まとめ
Step-by-Step Distillation = 2023年Hsieh et al. Google Cloud AI Research+UW発表のrationale-based LLM knowledge distillation。Industry-leading chain-of-thought rationales as additional supervision + Industry-leading teacher LLM generates rationales not just labels + Industry-leading multi-task learning label prediction + rationale prediction + Industry-leading 700× smaller than teacher possible (540B → 770M typical) + Industry-leading PaLM 540B teacher → T5 220M/770M/3B student + Industry-leading 770M student outperforms 540B PaLM teacher on 4/4 benchmarks + Industry-leading 12.5% training data efficiency + Industry-leading Cheng-Yu Hsieh + Chun-Liang Li + Chih-Kuan Yeh Google Cloud AI Research + UW + Industry-leading Step-by-Step Distillation GitHub google-research/distilling-step-by-step open source + Industry-leading rationale-based LLM distillation Google Research 2023 position確立。