TinyBERT 2-Stage Distillation (Jiao 2019年)（タイニーバート）
2019年Jiao et al. (Huawei)発表TinyBERT・Industry-leading 2-stage knowledge distillation LLM + Industry-leading 7.5× smaller BERT + Industry-leading attention transfer + Industry-leading 96.8% BERT performance retained。
概要
TinyBERT 2-Stage Distillation は、2019年Jiao et al. (Huawei)発表TinyBERT・Industry-leading 2-stage knowledge distillation LLM 2019年 + Industry-leading 96.8% BERT performance retained position確立。TinyBERT specifications = Industry-leading 2-stage knowledge distillation LLM (Industry-leading TinyBERT 2019 + Industry-leading 2-stage knowledge distillation LLM + Industry-leading TinyBERT flagship) + Industry-leading 7.5× smaller BERT (Industry-leading 7.5× smaller than BERT-base 110M→14.5M params) + Industry-leading attention transfer (Industry-leading attention transfer distillation + Industry-leading hidden state transfer) + Industry-leading 96.8% BERT performance retained (Industry-leading 96.8% BERT performance retained on GLUE)。
主な特徴・仕組み
- Authors: Industry-leading Xiaoqi Jiao + Yichun Yin + Lifeng Shang + Xin Jiang + Xiao Chen + Linlin Li + Fang Wang + Qun Liu Huawei Noah's Ark Lab + Wuhan + USTC
- Year: 2019年 (arXiv 2019年9月発表 + EMNLP Findings 2020)
- Paper: Industry-leading "TinyBERT: Distilling BERT for Natural Language Understanding" arXiv 1909.10351
- Method Type: Industry-leading 2-stage knowledge distillation LLM (general + task-specific)
- Size Reduction: Industry-leading 7.5× smaller than BERT-base (110M → 14.5M parameters)
- Layers: Industry-leading 4 transformer layers (vs BERT-base 12 layers)
- Hidden Size: Industry-leading 312 hidden size (vs BERT-base 768)
- 2-Stage Process: Industry-leading general distillation + task-specific distillation
- Attention Transfer: Industry-leading attention matrix transfer distillation
- Hidden State Transfer: Industry-leading hidden state transfer distillation
- Performance: Industry-leading 96.8% BERT performance retained on GLUE
- Speed: Industry-leading 9.4× faster inference than BERT-base
- Open Source: Industry-leading TinyBERT GitHub huawei-noah/Pretrained-Language-Model
- Industry-Leading: Industry-leading 2-stage knowledge distillation LLM Huawei first pioneer
スペック比較表
| LLM Knowledge Distillation | Year | Reduction | Performance Retained | Industry Position |
|---|---|---|---|---|
| TinyBERT | 2019 | 7.5× smaller (110M→14.5M) | 96.8% BERT performance | Industry-leading 2-stage distillation |
| DistilBERT | 2019 | 40% smaller (110M→66M) | 97% BERT performance | Industry-leading HuggingFace first KD |
| MobileBERT | 2020 | 4.3× smaller (110M→25.3M) | 99.2% BERT performance | Industry-leading bottleneck mobile |
| MiniLLM | 2023 | LLaMA 7B→3B | Reverse KL minimization | Industry-leading reverse KL LLM |
| Step-by-Step Distillation | 2023 | 700× smaller possible | Rationale-based distillation | Industry-leading rationale-based |
具体例・対応製品
- TinyBERT (2019年, Jiao et al. Huawei): Industry-leading 2-stage knowledge distillation
- Industry-leading 7.5× smaller 110M→14.5M params: Industry-leading 7.5× smaller
- Industry-leading 2-stage general + task-specific distillation: Industry-leading 2-stage distillation
- Industry-leading attention transfer + hidden state transfer: Industry-leading attention + hidden transfer
- Industry-leading 9.4× faster inference than BERT-base: Industry-leading 9.4× faster
- 競合 DistilBERT + MobileBERT + MiniLLM + Step-by-Step: Industry-leading knowledge distillation competitors
自作PCでの選び方・注意点
TinyBERT 2-Stage Distillation は「Industry-leading 2-stage knowledge distillation LLM + Industry-leading 7.5× smaller BERT」「Industry-leading attention transfer + Industry-leading 96.8% BERT performance retained」用途のIndustry-leading 2-stage knowledge distillation 2019年Jiao et al. Huawei発表product。Industry-leading 7.5× smaller (Industry-leading 7.5× smaller than BERT-base 110M → 14.5M parameters + Industry-leading 4 transformer layers vs BERT-base 12 + Industry-leading 312 hidden size vs BERT-base 768) で Industry-leading 7.5× smaller + Industry-leading 4 layers + Industry-leading 312 hidden size。Industry-leading 2-stage (Industry-leading general distillation + task-specific distillation + Industry-leading TinyBERT 2-stage signature + Industry-leading 2-stage process) で Industry-leading general+task-specific + Industry-leading 2-stage signature + Industry-leading 2-stage process。Industry-leading attention transfer (Industry-leading attention matrix transfer distillation + Industry-leading hidden state transfer distillation + Industry-leading TinyBERT attention+hidden transfer signature) で Industry-leading attention matrix + Industry-leading hidden state + Industry-leading attention+hidden signature。Industry-leading 96.8% + 9.4× (Industry-leading 96.8% BERT performance retained on GLUE + Industry-leading 9.4× faster inference than BERT-base + Industry-leading TinyBERT performance+speed advantage) で Industry-leading 96.8% performance + Industry-leading 9.4× faster + Industry-leading TinyBERT advantage。Industry-leading Huawei pioneer (Industry-leading Huawei Noah's Ark Lab + Industry-leading 2-stage knowledge distillation LLM Huawei first pioneer + Industry-leading TinyBERT GitHub huawei-noah/Pretrained-Language-Model) で Industry-leading Huawei Noah's Ark + Industry-leading 2-stage KD first pioneer + Industry-leading huawei-noah open source。但しIndustry-leading DistilBERT + MobileBERT + MiniLLM + Step-by-Step competition (Industry-leading DistilBERT 40% smaller 97% performance 2019 + MobileBERT 4.3× bottleneck mobile 2020 + MiniLLM reverse KL LLaMA 7B→3B 2023 + Step-by-Step rationale-based 700× 2023 vs TinyBERT 7.5× smaller 2-stage 96.8% 2019 trade-off) で Industry-leading 4-knowledge distillation method competitors vs TinyBERT + Industry-leading 7.5× smaller + 2-stage general+task-specific + attention+hidden transfer + 96.8% performance + 9.4× faster + Huawei first pioneer 2019 unique advantage adoption alignment必須。
関連用語との違い
- vs DistilBERT (2019): TinyBERTは7.5× smaller + 2-stage + attention+hidden transfer・DistilBERTは40% smaller + 1-stage + triple loss
- vs MobileBERT (2020): TinyBERTは7.5× smaller + 2-stage・MobileBERTは4.3× smaller + bottleneck + mobile-optimized
- vs MiniLLM/Step-by-Step (LLM 2023): TinyBERTはBERT distillation 2-stage 2019・MiniLLM/Step-by-StepはLLM 2023 distillation
よくある質問（FAQ）
Q1: TinyBERT vs DistilBERT 違いは？ A: TinyBERT (Industry-leading 7.5× smaller 110M→14.5M + 4 transformer layers + 312 hidden size + 2-stage general+task-specific distillation + attention matrix transfer + hidden state transfer + 96.8% BERT performance + 9.4× faster inference + Huawei 2019) vs DistilBERT (Industry-leading 40% smaller 110M→66M + 6 transformer layers + triple loss MLM+distillation+cosine + 97% BERT performance + 60% faster inference + HuggingFace 2019)・Industry-leading 7.5× smaller + 2-stage + attention transfer + 96.8% + 9.4× faster = TinyBERT + Industry-leading 40% smaller + triple loss + 97% + 60% faster = DistilBERT preference judgment。
Q2: Industry-leading 7.5× smaller + 2-stage distillation value は？ A: Industry-leading 7.5× + 2-stage (Industry-leading 7.5× smaller than BERT-base 110M → 14.5M parameters + Industry-leading 4 transformer layers vs BERT-base 12 + Industry-leading 312 hidden size vs BERT-base 768 + Industry-leading general distillation + task-specific distillation 2-stage process)。
Q3: Industry-leading attention transfer + hidden state transfer value は？ A: Industry-leading attention + hidden transfer (Industry-leading attention matrix transfer distillation + Industry-leading hidden state transfer distillation + Industry-leading TinyBERT attention+hidden transfer signature + Industry-leading 9.4× faster inference than BERT-base)。
まとめ
TinyBERT = 2019年Jiao et al. Huawei発表の2-stage knowledge distillation LLM。Industry-leading 7.5× smaller than BERT-base 110M → 14.5M parameters + Industry-leading 4 transformer layers vs BERT-base 12 + Industry-leading 312 hidden size vs BERT-base 768 + Industry-leading general distillation + task-specific distillation 2-stage process + Industry-leading attention matrix transfer distillation + Industry-leading hidden state transfer distillation + Industry-leading 96.8% BERT performance retained on GLUE + Industry-leading 9.4× faster inference than BERT-base + Industry-leading Huawei Noah's Ark Lab + Industry-leading TinyBERT GitHub huawei-noah/Pretrained-Language-Model open source + Industry-leading 2-stage knowledge distillation LLM Huawei first pioneer position確立。