LLM-Pruner Structural Pruning + LoRA (Ma 2023年)（エルエルエムプルーナー）
2023年Ma et al. (Singapore NUS)発表LLM-Pruner・Industry-leading gradient-based structured pruning LLM + Industry-leading LoRA recovery + Industry-leading 20% parameter reduction + Industry-leading LLaMA/Vicuna pruning。
概要
LLM-Pruner Structural Pruning + LoRA は、2023年Ma et al. (Singapore NUS)発表LLM-Pruner・Industry-leading gradient-based structured pruning LLM 2023年 + Industry-leading 20% parameter reduction + Industry-leading LLaMA/Vicuna pruning position確立。LLM-Pruner specifications = Industry-leading gradient-based structured pruning LLM (Industry-leading LLM-Pruner 2023 + Industry-leading gradient-based structured pruning LLM + Industry-leading LLM-Pruner flagship) + Industry-leading LoRA recovery (Industry-leading LoRA recovery post-pruning) + Industry-leading 20% parameter reduction (Industry-leading 20% parameter reduction) + Industry-leading LLaMA/Vicuna pruning (Industry-leading LLaMA + Vicuna structured pruning)。
主な特徴・仕組み
- Authors: Industry-leading Xinyin Ma + Gongfan Fang + Xinchao Wang Singapore National University (NUS) Learning and Vision Lab
- Year: 2023年 (arXiv 2023年5月発表 + NeurIPS 2023)
- Paper: Industry-leading "LLM-Pruner: On the Structural Pruning of Large Language Models" arXiv 2305.11627
- Method Type: Industry-leading gradient-based structured pruning LLM + LoRA recovery
- Sparsity Type: Industry-leading structured pruning (groups + dependencies)
- Key Innovation: Industry-leading dependency-aware structured pruning
- Pruning Metric: Industry-leading gradient-based importance estimation (1st + 2nd order Taylor)
- Dependency Detection: Industry-leading automatic dependency detection in LLM
- Recovery: Industry-leading LoRA recovery post-pruning (cheap finetuning)
- Reduction: Industry-leading 20% parameter reduction
- Target Models: Industry-leading LLaMA 7B/13B + Vicuna 7B/13B + ChatGLM-6B
- Open Source: Industry-leading LLM-Pruner GitHub horseee/LLM-Pruner
- Industry-Leading: Industry-leading gradient-based structured pruning LLM + LoRA recovery first pioneer
スペック比較表
| LLM Pruning Method | Year | Sparsity Type | Method | Industry Position |
|---|---|---|---|---|
| LLM-Pruner | 2023 | Structured 20% | Gradient-based structured pruning + LoRA recovery | Industry-leading gradient structured |
| ShearedLLaMA | 2023 | Structured 2.7B/1.3B | Targeted structured pruning + continued pretraining | Industry-leading structured ShearedLLaMA |
| SliceGPT | 2024 | Structured 25-30% | PCA-based slicing + orthogonal transformation | Industry-leading structured SliceGPT |
| SparseGPT | 2023 | Unstructured 50% + 2:4 + 4:8 | Second-order Hessian OBS derivative | Industry-leading 175B one-shot |
| Wanda | 2023 | Unstructured 50% + 2:4 | Weight×activation magnitude | Industry-leading lightweight no Hessian |
具体例・対応製品
- LLM-Pruner (2023年, Ma et al. Singapore NUS): Industry-leading gradient-based structured pruning
- Industry-leading gradient-based importance estimation: Industry-leading gradient-based estimation
- Industry-leading dependency-aware structured pruning: Industry-leading dependency-aware
- Industry-leading LoRA recovery post-pruning cheap finetuning: Industry-leading LoRA recovery
- Industry-leading LLaMA 7B/13B + Vicuna 7B/13B + ChatGLM-6B: Industry-leading LLaMA/Vicuna/ChatGLM
- 競合 SparseGPT + Wanda + ShearedLLaMA + SliceGPT: Industry-leading pruning competitors
自作PCでの選び方・注意点
LLM-Pruner Structural Pruning + LoRA は「Industry-leading gradient-based structured pruning LLM + Industry-leading LoRA recovery」「Industry-leading 20% parameter reduction + Industry-leading LLaMA/Vicuna pruning」用途のIndustry-leading gradient-based structured pruning 2023年Ma et al. Singapore NUS発表product。Industry-leading gradient-based (Industry-leading gradient-based importance estimation 1st + 2nd order Taylor + Industry-leading LLM-Pruner gradient signature + Industry-leading 1st + 2nd order Taylor importance) で Industry-leading gradient-based + Industry-leading LLM-Pruner signature + Industry-leading 1st+2nd order Taylor。Industry-leading dependency-aware (Industry-leading dependency-aware structured pruning + Industry-leading automatic dependency detection in LLM + Industry-leading LLM-Pruner dependency signature) で Industry-leading dependency-aware + Industry-leading automatic detection + Industry-leading LLM-Pruner signature。Industry-leading LoRA recovery (Industry-leading LoRA recovery post-pruning cheap finetuning + Industry-leading LLM-Pruner LoRA recovery signature + Industry-leading LoRA cheap recovery) で Industry-leading LoRA recovery + Industry-leading LLM-Pruner LoRA + Industry-leading cheap recovery。Industry-leading 20% reduction (Industry-leading 20% parameter reduction + Industry-leading LLaMA 7B/13B + Vicuna 7B/13B + ChatGLM-6B + Industry-leading LLM-Pruner first pioneer 2023) で Industry-leading 20% reduction + Industry-leading LLaMA/Vicuna/ChatGLM + Industry-leading first pioneer。Industry-leading open source (Industry-leading LLM-Pruner GitHub horseee/LLM-Pruner + Industry-leading gradient-based structured pruning LLM + LoRA recovery first pioneer) で Industry-leading horseee/LLM-Pruner open source + Industry-leading gradient structured + LoRA recovery first pioneer。但しIndustry-leading SparseGPT + Wanda + ShearedLLaMA + SliceGPT competition (Industry-leading SparseGPT Hessian unstructured 50% 2023 + Wanda W×A unstructured 2023 + ShearedLLaMA targeted structured + 50B continued 2023 + SliceGPT PCA structured 25-30% 2024 vs LLM-Pruner gradient structured 20% + LoRA recovery 2023 trade-off) で Industry-leading 4-pruning method competitors vs LLM-Pruner + Industry-leading gradient-based + dependency-aware + LoRA recovery + 20% + LLaMA/Vicuna/ChatGLM + first pioneer 2023 unique advantage adoption alignment必須。
関連用語との違い
- vs ShearedLLaMA (Targeted Structured 2023): LLM-PrunerはGradient + LoRA recovery + 20%・ShearedLLaMAはtargeted + continued pretraining 50B + 7B→2.7B
- vs SliceGPT (PCA Structured 2024): LLM-PrunerはGradient + LoRA + 20%・SliceGPTはPCA + orthogonal + 25-30% + no retraining
- vs SparseGPT/Wanda (Unstructured 50%): LLM-Prunerはstructured 20% + LoRA・SparseGPT/Wandaはunstructured 50%
よくある質問（FAQ）
Q1: LLM-Pruner vs ShearedLLaMA 違いは？ A: LLM-Pruner (Industry-leading gradient-based importance estimation + 1st + 2nd order Taylor + dependency-aware structured pruning + automatic dependency detection + LoRA recovery post-pruning cheap finetuning + 20% parameter reduction + LLaMA + Vicuna + ChatGLM-6B + Singapore NUS 2023) vs ShearedLLaMA (Industry-leading targeted structured pruning + layer + head + FFN dim + 50B tokens continued pretraining + 3% cost + LLaMA-2 7B → 2.7B/1.3B + outperforms Pythia + Princeton 2023)・Industry-leading gradient + dependency-aware + LoRA + 20% = LLM-Pruner + Industry-leading targeted + 50B continued + 7B→2.7B = ShearedLLaMA preference judgment。
Q2: Industry-leading gradient-based + dependency-aware value は？ A: Industry-leading gradient + dependency (Industry-leading gradient-based importance estimation 1st + 2nd order Taylor + Industry-leading dependency-aware structured pruning + Industry-leading automatic dependency detection in LLM + Industry-leading LLM-Pruner gradient + dependency signature)。
Q3: Industry-leading LoRA recovery + LLaMA/Vicuna pruning value は？ A: Industry-leading LoRA + LLaMA/Vicuna (Industry-leading LoRA recovery post-pruning cheap finetuning + Industry-leading LLM-Pruner LoRA recovery signature + Industry-leading LLaMA 7B/13B + Vicuna 7B/13B + ChatGLM-6B + Industry-leading 20% parameter reduction)。
まとめ
LLM-Pruner = 2023年Ma et al. Singapore NUS発表のgradient-based structured pruning LLM + LoRA recovery。Industry-leading gradient-based importance estimation 1st + 2nd order Taylor + Industry-leading dependency-aware structured pruning + Industry-leading automatic dependency detection in LLM + Industry-leading LoRA recovery post-pruning cheap finetuning + Industry-leading 20% parameter reduction + Industry-leading LLaMA 7B/13B + Vicuna 7B/13B + ChatGLM-6B pruning + Industry-leading LLM-Pruner GitHub horseee/LLM-Pruner open source + Industry-leading gradient-based structured pruning LLM + LoRA recovery first pioneer 2023 position確立。