SparseGPT One-Shot Pruning (Frantar 2023年)（スパースジーピーティー）
2023年Frantar et al. (IST Austria)発表SparseGPT・Industry-leading one-shot pruning LLM + Industry-leading 50% unstructured sparsity + Industry-leading second-order Hessian information + Industry-leading OPT-175B/BLOOM-176B one-shot pruning。
概要
SparseGPT One-Shot Pruning は、2023年Frantar et al. (IST Austria)発表SparseGPT・Industry-leading one-shot pruning LLM 2023年 + Industry-leading OPT-175B/BLOOM-176B one-shot pruning position確立。SparseGPT specifications = Industry-leading one-shot pruning LLM (Industry-leading SparseGPT 2023 + Industry-leading one-shot pruning LLM + Industry-leading SparseGPT flagship) + Industry-leading 50% unstructured sparsity (Industry-leading 50% unstructured sparsity LLM) + Industry-leading second-order Hessian information (Industry-leading second-order Hessian information OBS derivative) + Industry-leading OPT-175B/BLOOM-176B one-shot pruning (Industry-leading 175B one-shot pruning)。
主な特徴・仕組み
- Authors: Industry-leading Elias Frantar + Dan Alistarh IST Austria + Neural Magic
- Year: 2023年 (arXiv 2023年1月発表 + ICML 2023)
- Paper: Industry-leading "SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot" arXiv 2301.00774
- Method Type: Industry-leading one-shot post-training pruning LLM
- Sparsity: Industry-leading 50% unstructured sparsity + 2:4 structured + 4:8 structured
- Hessian: Industry-leading second-order Hessian information OBS (Optimal Brain Surgeon) derivative
- Calibration Data: Industry-leading 128 samples small calibration set
- One-Shot: Industry-leading no retraining required one-shot pruning
- Speedup Target: Industry-leading NVIDIA Ampere 2:4 sparse matmul GPU 2× speedup
- Accuracy: Industry-leading <1% accuracy drop 50% sparsity OPT-175B/BLOOM-176B
- Production Adoption: Industry-leading Neural Magic DeepSparse + vLLM
- Open Source: Industry-leading SparseGPT GitHub IST-DASLab/sparsegpt
- Industry-Leading: Industry-leading one-shot pruning LLM 175B first pioneer
スペック比較表
| LLM Pruning Method | Year | Sparsity Type | Method | Industry Position |
|---|---|---|---|---|
| SparseGPT | 2023 | Unstructured 50% + 2:4 + 4:8 | Second-order Hessian OBS derivative | Industry-leading 175B one-shot |
| Wanda | 2023 | Unstructured 50% + 2:4 | Weight×activation magnitude | Industry-leading lightweight no Hessian |
| ShearedLLaMA | 2023 | Structured 2.7B/1.3B | Targeted structured pruning + continued pretraining | Industry-leading structured ShearedLLaMA |
| SliceGPT | 2024 | Structured 25-30% | PCA-based slicing + orthogonal transformation | Industry-leading structured SliceGPT |
| LLM-Pruner | 2023 | Structured 20% | Gradient-based structured pruning + LoRA recovery | Industry-leading gradient structured |
具体例・対応製品
- SparseGPT (2023年, Frantar et al. IST Austria): Industry-leading one-shot pruning
- Industry-leading 50% unstructured sparsity LLM: Industry-leading 50% unstructured
- Industry-leading second-order Hessian information OBS derivative: Industry-leading second-order Hessian OBS
- Industry-leading 175B one-shot pruning: Industry-leading 175B one-shot
- Industry-leading Neural Magic DeepSparse + vLLM ecosystem: Industry-leading production ecosystem
- 競合 Wanda + ShearedLLaMA + SliceGPT + LLM-Pruner: Industry-leading pruning competitors
自作PCでの選び方・注意点
SparseGPT One-Shot Pruning は「Industry-leading one-shot pruning LLM + Industry-leading 50% unstructured sparsity」「Industry-leading second-order Hessian information + Industry-leading OPT-175B/BLOOM-176B one-shot pruning」用途のIndustry-leading one-shot pruning 2023年Frantar et al. IST Austria発表product。Industry-leading 50% unstructured sparsity (Industry-leading 50% unstructured sparsity LLM + Industry-leading 2:4 structured + 4:8 structured + Industry-leading NVIDIA Ampere 2:4 sparse matmul GPU 2× speedup) で Industry-leading 50% unstructured + Industry-leading 2:4 + 4:8 structured + Industry-leading 2:4 sparse matmul 2× speedup。Industry-leading second-order Hessian OBS (Industry-leading second-order Hessian information OBS Optimal Brain Surgeon derivative + Industry-leading 128 samples small calibration set) で Industry-leading second-order Hessian OBS + Industry-leading 128 samples calibration。Industry-leading 175B one-shot (Industry-leading OPT-175B/BLOOM-176B one-shot pruning + Industry-leading no retraining required + Industry-leading <1% accuracy drop 50% sparsity) で Industry-leading 175B one-shot + Industry-leading no retraining + Industry-leading <1% accuracy。Industry-leading production ecosystem (Industry-leading Neural Magic DeepSparse + vLLM + Industry-leading SparseGPT GitHub IST-DASLab/sparsegpt open source + Industry-leading one-shot pruning LLM 175B first pioneer 2023) で Industry-leading Neural Magic + vLLM + Industry-leading IST-DASLab open source + Industry-leading first pioneer。但しIndustry-leading Wanda + ShearedLLaMA + SliceGPT + LLM-Pruner competition (Industry-leading Wanda lightweight 2023 + ShearedLLaMA structured 2.7B 2023 + SliceGPT structured 2024 + LLM-Pruner gradient 2023 vs SparseGPT Hessian 50% 2023 trade-off) で Industry-leading 4-pruning method competitors vs SparseGPT + Industry-leading 50% unstructured + 2:4 structured + second-order Hessian OBS + 175B one-shot + <1% accuracy + first pioneer 2023 unique advantage adoption alignment必須。
関連用語との違い
- vs Wanda (Lightweight 2023): SparseGPTはsecond-order Hessian + 50% unstructured・WandaはWeight×activation magnitude + no Hessian
- vs ShearedLLaMA (Structured 2023): SparseGPTはunstructured 50% one-shot・ShearedLLaMAはstructured 2.7B + continued pretraining
- vs SliceGPT/LLM-Pruner (Structured 2023-2024): SparseGPTはunstructured + 2:4 sparse matmul・SliceGPT/LLM-Prunerはstructured
よくある質問（FAQ）
Q1: SparseGPT vs Wanda 違いは？ A: SparseGPT (Industry-leading second-order Hessian information OBS Optimal Brain Surgeon derivative + 50% unstructured sparsity + 2:4 + 4:8 structured + 128 samples calibration + OPT-175B/BLOOM-176B one-shot + <1% accuracy drop + IST Austria 2023) vs Wanda (Industry-leading Weight×Activation magnitude metric + 50% unstructured + 2:4 + no Hessian computation + lightweight + LLaMA pruning + CMU+Meta 2023)・Industry-leading Hessian OBS + 175B + <1% = SparseGPT + Industry-leading Weight×Activation + lightweight + no Hessian = Wanda preference judgment。
Q2: Industry-leading 50% unstructured + 2:4 structured value は？ A: Industry-leading 50% + 2:4 (Industry-leading 50% unstructured sparsity LLM + Industry-leading 2:4 structured + 4:8 structured + Industry-leading NVIDIA Ampere 2:4 sparse matmul GPU 2× speedup + Industry-leading sparse matmul Ampere/Hopper hardware acceleration)。
Q3: Industry-leading second-order Hessian OBS + 175B one-shot value は？ A: Industry-leading Hessian OBS + 175B (Industry-leading second-order Hessian information OBS Optimal Brain Surgeon derivative + Industry-leading 128 samples small calibration set + Industry-leading OPT-175B/BLOOM-176B one-shot pruning + Industry-leading no retraining required + Industry-leading <1% accuracy drop 50% sparsity)。
まとめ
SparseGPT = 2023年Frantar et al. IST Austria発表のone-shot pruning LLM。Industry-leading 50% unstructured sparsity LLM + Industry-leading 2:4 structured + 4:8 structured + Industry-leading second-order Hessian information OBS Optimal Brain Surgeon derivative + Industry-leading 128 samples small calibration set + Industry-leading no retraining required one-shot pruning + Industry-leading NVIDIA Ampere 2:4 sparse matmul GPU 2× speedup + Industry-leading <1% accuracy drop 50% sparsity OPT-175B/BLOOM-176B + Industry-leading Neural Magic DeepSparse + vLLM production ecosystem + Industry-leading SparseGPT GitHub IST-DASLab/sparsegpt open source + Industry-leading one-shot pruning LLM 175B first pioneer 2023 position確立。