MobileBERT Bottleneck Distillation (Sun 2020年)（モバイルバート）
2020年Sun et al. (Google+CMU)発表MobileBERT・Industry-leading mobile-optimized knowledge distillation LLM + Industry-leading 4.3× smaller BERT + Industry-leading bottleneck architecture + Industry-leading 99.2% BERT performance retained。
概要
MobileBERT Bottleneck Distillation は、2020年Sun et al. (Google+CMU)発表MobileBERT・Industry-leading mobile-optimized knowledge distillation LLM 2020年 + Industry-leading 99.2% BERT performance retained position確立。MobileBERT specifications = Industry-leading mobile-optimized knowledge distillation LLM (Industry-leading MobileBERT 2020 + Industry-leading mobile-optimized knowledge distillation LLM + Industry-leading MobileBERT flagship) + Industry-leading 4.3× smaller BERT (Industry-leading 4.3× smaller than BERT-base 110M→25.3M params) + Industry-leading bottleneck architecture (Industry-leading inverted-bottleneck architecture mobile-optimized) + Industry-leading 99.2% BERT performance retained (Industry-leading 99.2% BERT-base performance retained on GLUE highest)。
主な特徴・仕組み
- Authors: Industry-leading Zhiqing Sun + Hongkun Yu + Xiaodan Song + Renjie Liu + Yiming Yang + Denny Zhou Google Research + CMU
- Year: 2020年 (arXiv 2020年4月発表 + ACL 2020)
- Paper: Industry-leading "MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices" arXiv 2004.02984
- Method Type: Industry-leading mobile-optimized knowledge distillation LLM
- Size Reduction: Industry-leading 4.3× smaller than BERT-base (110M → 25.3M parameters)
- Architecture: Industry-leading inverted-bottleneck transformer architecture
- Layers: Industry-leading 24 transformer layers (deep but narrow)
- Hidden Size: Industry-leading 128 (bottleneck) / 512 (inverted-bottleneck)
- Mobile-Optimized: Industry-leading mobile inference optimized
- Performance: Industry-leading 99.2% BERT-base performance retained on GLUE (highest in 5-tier)
- Speed: Industry-leading 4× faster on Pixel 4
- Teacher: Industry-leading IB-BERT (Inverted-Bottleneck BERT) teacher
- Open Source: Industry-leading MobileBERT GitHub google-research/google-research/mobilebert
- Industry-Leading: Industry-leading mobile-optimized knowledge distillation LLM Google+CMU
スペック比較表
| LLM Knowledge Distillation | Year | Reduction | Performance Retained | Industry Position |
|---|---|---|---|---|
| MobileBERT | 2020 | 4.3× smaller (110M→25.3M) | 99.2% BERT performance highest | Industry-leading bottleneck mobile |
| TinyBERT | 2019 | 7.5× smaller (110M→14.5M) | 96.8% BERT performance | Industry-leading 2-stage distillation |
| DistilBERT | 2019 | 40% smaller (110M→66M) | 97% BERT performance | Industry-leading HuggingFace first KD |
| MiniLLM | 2023 | LLaMA 7B→3B | Reverse KL minimization | Industry-leading reverse KL LLM |
| Step-by-Step Distillation | 2023 | 700× smaller possible | Rationale-based distillation | Industry-leading rationale-based |
具体例・対応製品
- MobileBERT (2020年, Sun et al. Google+CMU): Industry-leading mobile-optimized knowledge distillation
- Industry-leading 4.3× smaller 110M→25.3M params: Industry-leading 4.3× smaller
- Industry-leading inverted-bottleneck transformer architecture: Industry-leading inverted-bottleneck
- Industry-leading 99.2% BERT-base performance retained highest: Industry-leading 99.2% highest
- Industry-leading 4× faster on Pixel 4 mobile: Industry-leading 4× faster Pixel 4
- 競合 DistilBERT + TinyBERT + MiniLLM + Step-by-Step: Industry-leading knowledge distillation competitors
自作PCでの選び方・注意点
MobileBERT Bottleneck Distillation は「Industry-leading mobile-optimized knowledge distillation LLM + Industry-leading 4.3× smaller BERT」「Industry-leading bottleneck architecture + Industry-leading 99.2% BERT performance retained」用途のIndustry-leading mobile-optimized knowledge distillation 2020年Sun et al. Google+CMU発表product。Industry-leading 4.3× smaller (Industry-leading 4.3× smaller than BERT-base 110M → 25.3M parameters + Industry-leading 24 transformer layers deep but narrow + Industry-leading 128 bottleneck / 512 inverted-bottleneck hidden size) で Industry-leading 4.3× smaller + Industry-leading 24 layers deep+narrow + Industry-leading 128/512 bottleneck。Industry-leading inverted-bottleneck (Industry-leading inverted-bottleneck transformer architecture + Industry-leading IB-BERT Inverted-Bottleneck BERT teacher + Industry-leading MobileBERT bottleneck signature) で Industry-leading inverted-bottleneck + Industry-leading IB-BERT teacher + Industry-leading bottleneck signature。Industry-leading 99.2% highest (Industry-leading 99.2% BERT-base performance retained on GLUE highest in 5-tier + Industry-leading MobileBERT performance highest + Industry-leading mobile-optimized highest performance) で Industry-leading 99.2% highest + Industry-leading performance highest + Industry-leading mobile-optimized highest。Industry-leading mobile-optimized (Industry-leading mobile inference optimized + Industry-leading 4× faster on Pixel 4 + Industry-leading mobile-optimized BERT signature + Industry-leading resource-limited devices target) で Industry-leading mobile inference + Industry-leading 4× faster Pixel 4 + Industry-leading mobile-optimized + Industry-leading resource-limited target。Industry-leading Google+CMU (Industry-leading Zhiqing Sun + Hongkun Yu + Xiaodan Song + Denny Zhou Google Research + CMU + Industry-leading MobileBERT GitHub google-research/google-research/mobilebert + Industry-leading mobile-optimized KD LLM) で Industry-leading Sun+Yu+Song+Zhou Google+CMU + Industry-leading google-research open source + Industry-leading mobile-optimized KD LLM。但しIndustry-leading DistilBERT + TinyBERT + MiniLLM + Step-by-Step competition (Industry-leading DistilBERT 40% 97% 2019 + TinyBERT 7.5× 96.8% 2019 + MiniLLM reverse KL LLaMA 7B→3B 2023 + Step-by-Step rationale-based 700× 2023 vs MobileBERT 4.3× 99.2% bottleneck mobile 2020 trade-off) で Industry-leading 4-knowledge distillation method competitors vs MobileBERT + Industry-leading 4.3× smaller + inverted-bottleneck + 24 layers deep+narrow + 99.2% performance highest + mobile-optimized + 4× faster Pixel 4 + Google+CMU 2020 unique advantage adoption alignment必須。
関連用語との違い
- vs DistilBERT (2019): MobileBERTはbottleneck + 24 layers deep+narrow + 99.2% highest・DistilBERTは6 layers + triple loss + 97%
- vs TinyBERT (2019): MobileBERTはbottleneck + mobile-optimized + 99.2%・TinyBERTは2-stage + attention transfer + 96.8%
- vs MiniLLM/Step-by-Step (LLM 2023): MobileBERTはBERT mobile-optimized 2020・MiniLLM/Step-by-StepはLLM 2023 distillation
よくある質問（FAQ）
Q1: MobileBERT vs DistilBERT/TinyBERT 違いは？ A: MobileBERT (Industry-leading 4.3× smaller 110M→25.3M + 24 transformer layers deep but narrow + 128 bottleneck / 512 inverted-bottleneck hidden size + inverted-bottleneck architecture + IB-BERT teacher + 99.2% BERT performance highest + 4× faster on Pixel 4 + mobile-optimized + Google+CMU 2020) vs DistilBERT (40% smaller + 6 layers + triple loss + 97% + HuggingFace 2019) vs TinyBERT (7.5× smaller + 4 layers + 2-stage + attention transfer + 96.8% + Huawei 2019)・Industry-leading bottleneck + 24 deep+narrow + 99.2% + mobile + 4× Pixel = MobileBERT preference judgment。
Q2: Industry-leading inverted-bottleneck + 24 layers deep+narrow value は？ A: Industry-leading inverted-bottleneck + 24 deep+narrow (Industry-leading inverted-bottleneck transformer architecture + Industry-leading 24 transformer layers deep but narrow + Industry-leading 128 bottleneck / 512 inverted-bottleneck hidden size + Industry-leading IB-BERT Inverted-Bottleneck BERT teacher)。
Q3: Industry-leading 99.2% highest + mobile-optimized value は？ A: Industry-leading 99.2% + mobile (Industry-leading 99.2% BERT-base performance retained on GLUE highest in 5-tier + Industry-leading mobile inference optimized + Industry-leading 4× faster on Pixel 4 + Industry-leading resource-limited devices target)。
まとめ
MobileBERT = 2020年Sun et al. Google+CMU発表のmobile-optimized knowledge distillation LLM。Industry-leading 4.3× smaller than BERT-base 110M → 25.3M parameters + Industry-leading inverted-bottleneck transformer architecture + Industry-leading 24 transformer layers deep but narrow + Industry-leading 128 bottleneck / 512 inverted-bottleneck hidden size + Industry-leading IB-BERT Inverted-Bottleneck BERT teacher + Industry-leading 99.2% BERT-base performance retained on GLUE highest in 5-tier + Industry-leading 4× faster on Pixel 4 mobile + Industry-leading mobile inference optimized + Industry-leading resource-limited devices target + Industry-leading MobileBERT GitHub google-research/google-research/mobilebert open source + Industry-leading mobile-optimized knowledge distillation LLM Google+CMU 2020 position確立。