ShearedLLaMA Structured Pruning (Xia 2023年)（シェアードラマ）
2023年Xia et al. (Princeton)発表ShearedLLaMA・Industry-leading structured pruning LLM + Industry-leading targeted structured pruning + Industry-leading continued pretraining + Industry-leading LLaMA-2 7B→2.7B/1.3B structured pruning。
概要
ShearedLLaMA Structured Pruning は、2023年Xia et al. (Princeton)発表ShearedLLaMA・Industry-leading structured pruning LLM 2023年 + Industry-leading LLaMA-2 7B→2.7B/1.3B structured pruning position確立。ShearedLLaMA specifications = Industry-leading structured pruning LLM (Industry-leading ShearedLLaMA 2023 + Industry-leading structured pruning LLM + Industry-leading ShearedLLaMA flagship) + Industry-leading targeted structured pruning (Industry-leading targeted structured pruning + Industry-leading layer/head/intermediate dimension pruning) + Industry-leading continued pretraining (Industry-leading continued pretraining 50B tokens post-pruning) + Industry-leading LLaMA-2 7B→2.7B/1.3B (Industry-leading 2.7B/1.3B compressed models)。
主な特徴・仕組み
- Authors: Industry-leading Mengzhou Xia + Tianyu Gao + Zhiyuan Zeng + Danqi Chen Princeton + Tsinghua
- Year: 2023年 (arXiv 2023年10月発表 + ICLR 2024)
- Paper: Industry-leading "Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning" arXiv 2310.06694
- Method Type: Industry-leading structured pruning + continued pretraining LLM
- Sparsity Type: Industry-leading structured pruning (layer/head/intermediate dim)
- Target Models: Industry-leading LLaMA-2 7B → ShearedLLaMA 2.7B + 1.3B
- Pruning Targets: Industry-leading layer + multi-head attention head + intermediate FFN dimension
- Continued Pretraining: Industry-leading 50B tokens continued pretraining post-pruning
- Cost Efficiency: Industry-leading 3% cost of training from scratch
- Performance: Industry-leading ShearedLLaMA 2.7B outperforms Pythia 2.8B + INCITE 3B
- Open Source: Industry-leading ShearedLLaMA GitHub princeton-nlp/LLM-Shearing
- Industry-Leading: Industry-leading structured pruning + continued pretraining LLM
スペック比較表
| LLM Pruning Method | Year | Sparsity Type | Method | Industry Position |
|---|---|---|---|---|
| ShearedLLaMA | 2023 | Structured 2.7B/1.3B | Targeted structured pruning + continued pretraining | Industry-leading structured ShearedLLaMA |
| SparseGPT | 2023 | Unstructured 50% + 2:4 + 4:8 | Second-order Hessian OBS derivative | Industry-leading 175B one-shot |
| Wanda | 2023 | Unstructured 50% + 2:4 | Weight×activation magnitude | Industry-leading lightweight no Hessian |
| SliceGPT | 2024 | Structured 25-30% | PCA-based slicing + orthogonal transformation | Industry-leading structured SliceGPT |
| LLM-Pruner | 2023 | Structured 20% | Gradient-based structured pruning + LoRA recovery | Industry-leading gradient structured |
具体例・対応製品
- ShearedLLaMA (2023年, Xia et al. Princeton): Industry-leading structured pruning
- Industry-leading targeted structured pruning: Industry-leading targeted structured
- Industry-leading continued pretraining 50B tokens: Industry-leading continued pretraining 50B
- Industry-leading LLaMA-2 7B → ShearedLLaMA 2.7B + 1.3B: Industry-leading 7B→2.7B/1.3B
- Industry-leading 3% cost of training from scratch: Industry-leading 3% cost efficiency
- 競合 SparseGPT + Wanda + SliceGPT + LLM-Pruner: Industry-leading pruning competitors
自作PCでの選び方・注意点
ShearedLLaMA Structured Pruning は「Industry-leading structured pruning LLM + Industry-leading targeted structured pruning」「Industry-leading continued pretraining + Industry-leading LLaMA-2 7B→2.7B/1.3B structured pruning」用途のIndustry-leading structured pruning + continued pretraining 2023年Xia et al. Princeton発表product。Industry-leading targeted structured (Industry-leading targeted structured pruning + Industry-leading layer + multi-head attention head + intermediate FFN dimension pruning + Industry-leading ShearedLLaMA targeted signature) で Industry-leading targeted structured + Industry-leading layer + head + FFN dim + Industry-leading ShearedLLaMA signature。Industry-leading continued pretraining (Industry-leading 50B tokens continued pretraining post-pruning + Industry-leading 3% cost of training from scratch + Industry-leading dynamic batch loading) で Industry-leading 50B continued + Industry-leading 3% cost + Industry-leading dynamic batch loading。Industry-leading LLaMA 7B→2.7B/1.3B (Industry-leading LLaMA-2 7B → ShearedLLaMA 2.7B + 1.3B + Industry-leading ShearedLLaMA 2.7B outperforms Pythia 2.8B + INCITE 3B + Industry-leading ShearedLLaMA flagship) で Industry-leading 7B→2.7B/1.3B + Industry-leading outperforms Pythia/INCITE + Industry-leading ShearedLLaMA flagship。Industry-leading open source (Industry-leading ShearedLLaMA GitHub princeton-nlp/LLM-Shearing + Industry-leading structured pruning + continued pretraining LLM open source) で Industry-leading princeton-nlp open source + Industry-leading structured + continued pretraining。但しIndustry-leading SparseGPT + Wanda + SliceGPT + LLM-Pruner competition (Industry-leading SparseGPT Hessian unstructured 50% 2023 + Wanda W×A unstructured 50% 2023 + SliceGPT structured 25-30% 2024 + LLM-Pruner gradient structured 20% 2023 vs ShearedLLaMA structured 2.7B + continued pretraining 2023 trade-off) で Industry-leading 4-pruning method competitors vs ShearedLLaMA + Industry-leading targeted structured + continued pretraining 50B + 3% cost + 7B→2.7B/1.3B + outperforms Pythia/INCITE unique advantage adoption alignment必須。
関連用語との違い
- vs SparseGPT/Wanda (Unstructured 50%): ShearedLLaMAはstructured 2.7B + continued pretraining・SparseGPT/Wandaはunstructured 50% + no continued pretraining
- vs SliceGPT (Structured 2024): ShearedLLaMAはtargeted + continued pretraining 50B・SliceGPTはPCA slicing + no continued pretraining
- vs LLM-Pruner (Structured 2023): ShearedLLaMAはcontinued pretraining 50B・LLM-PrunerはLoRA recovery
よくある質問（FAQ）
Q1: ShearedLLaMA vs LLM-Pruner 違いは？ A: ShearedLLaMA (Industry-leading targeted structured pruning + layer + multi-head attention head + intermediate FFN dimension + 50B tokens continued pretraining post-pruning + 3% cost of training from scratch + LLaMA-2 7B → ShearedLLaMA 2.7B + 1.3B + outperforms Pythia 2.8B + INCITE 3B + Princeton 2023) vs LLM-Pruner (Industry-leading gradient-based structured pruning + 20% parameter reduction + LoRA recovery post-pruning + LLaMA + Vicuna + Singapore NUS 2023)・Industry-leading targeted + 50B continued + 7B→2.7B + outperforms Pythia = ShearedLLaMA + Industry-leading gradient + 20% + LoRA recovery + LLaMA = LLM-Pruner preference judgment。
Q2: Industry-leading targeted structured + continued pretraining 50B value は？ A: Industry-leading targeted + 50B continued (Industry-leading targeted structured pruning + Industry-leading layer + multi-head attention head + intermediate FFN dimension pruning + Industry-leading 50B tokens continued pretraining post-pruning + Industry-leading 3% cost of training from scratch)。
Q3: Industry-leading LLaMA-2 7B→2.7B/1.3B + outperforms baselines value は？ A: Industry-leading 7B→2.7B/1.3B + outperforms (Industry-leading LLaMA-2 7B → ShearedLLaMA 2.7B + 1.3B + Industry-leading ShearedLLaMA 2.7B outperforms Pythia 2.8B + INCITE 3B + Industry-leading ShearedLLaMA structured pruning + continued pretraining LLM)。
まとめ
ShearedLLaMA = 2023年Xia et al. Princeton発表のstructured pruning + continued pretraining LLM。Industry-leading targeted structured pruning + Industry-leading layer + multi-head attention head + intermediate FFN dimension pruning + Industry-leading 50B tokens continued pretraining post-pruning + Industry-leading 3% cost of training from scratch + Industry-leading dynamic batch loading + Industry-leading LLaMA-2 7B → ShearedLLaMA 2.7B + 1.3B + Industry-leading ShearedLLaMA 2.7B outperforms Pythia 2.8B + INCITE 3B + Industry-leading ShearedLLaMA GitHub princeton-nlp/LLM-Shearing open source + Industry-leading structured pruning + continued pretraining LLM position確立。