Wanda Weight×Activation Pruning (Sun 2023年)（ワンダ）
2023年Sun et al. (CMU+Meta)発表Wanda・Industry-leading lightweight pruning LLM + Industry-leading Weight×Activation magnitude metric + Industry-leading no Hessian computation + Industry-leading LLaMA pruning。
概要
Wanda Weight×Activation Pruning は、2023年Sun et al. (CMU+Meta)発表Wanda (Weights and Activations)・Industry-leading lightweight pruning LLM 2023年 + Industry-leading LLaMA pruning position確立。Wanda specifications = Industry-leading lightweight pruning LLM (Industry-leading Wanda 2023 + Industry-leading lightweight pruning LLM + Industry-leading Wanda flagship) + Industry-leading Weight×Activation magnitude metric (Industry-leading W×A metric pruning) + Industry-leading no Hessian computation (Industry-leading no second-order Hessian + Industry-leading lightweight) + Industry-leading LLaMA pruning (Industry-leading LLaMA 7B-70B pruning)。
主な特徴・仕組み
- Authors: Industry-leading Mingjie Sun + Zhuang Liu + Anna Bair + J. Zico Kolter CMU + Meta AI + Bosch Center
- Year: 2023年 (arXiv 2023年6月発表 + ICLR 2024)
- Paper: Industry-leading "A Simple and Effective Pruning Approach for Large Language Models" arXiv 2306.11695
- Method Type: Industry-leading lightweight post-training pruning LLM
- Pruning Metric: Industry-leading Weight×Activation (W×A) magnitude metric
- Sparsity: Industry-leading 50% unstructured + 2:4 structured
- Key Innovation: Industry-leading no Hessian computation + no weight update required
- Simplicity: Industry-leading simpler than SparseGPT (no inverse Hessian)
- Speed: Industry-leading 300× faster pruning than SparseGPT
- Accuracy: Industry-leading competitive with SparseGPT 50% sparsity
- Target Model: Industry-leading LLaMA 7B/13B/30B/65B + LLaMA-2 7B-70B
- Open Source: Industry-leading Wanda GitHub locuslab/wanda
- Industry-Leading: Industry-leading lightweight pruning LLM Weight×Activation
スペック比較表
| LLM Pruning Method | Year | Sparsity Type | Method | Industry Position |
|---|---|---|---|---|
| Wanda | 2023 | Unstructured 50% + 2:4 | Weight×activation magnitude | Industry-leading lightweight no Hessian |
| SparseGPT | 2023 | Unstructured 50% + 2:4 + 4:8 | Second-order Hessian OBS derivative | Industry-leading 175B one-shot |
| ShearedLLaMA | 2023 | Structured 2.7B/1.3B | Targeted structured pruning + continued pretraining | Industry-leading structured ShearedLLaMA |
| SliceGPT | 2024 | Structured 25-30% | PCA-based slicing + orthogonal transformation | Industry-leading structured SliceGPT |
| LLM-Pruner | 2023 | Structured 20% | Gradient-based structured pruning + LoRA recovery | Industry-leading gradient structured |
具体例・対応製品
- Wanda (2023年, Sun et al. CMU+Meta): Industry-leading lightweight pruning
- Industry-leading Weight×Activation (W×A) magnitude metric: Industry-leading W×A magnitude
- Industry-leading no Hessian computation + no weight update required: Industry-leading no Hessian + no weight update
- Industry-leading 300× faster pruning than SparseGPT: Industry-leading 300× faster pruning
- Industry-leading LLaMA 7B-70B + LLaMA-2 7B-70B pruning: Industry-leading LLaMA pruning
- 競合 SparseGPT + ShearedLLaMA + SliceGPT + LLM-Pruner: Industry-leading pruning competitors
自作PCでの選び方・注意点
Wanda Weight×Activation Pruning は「Industry-leading lightweight pruning LLM + Industry-leading Weight×Activation magnitude metric」「Industry-leading no Hessian computation + Industry-leading LLaMA pruning」用途のIndustry-leading lightweight pruning 2023年Sun et al. CMU+Meta発表product。Industry-leading W×A metric (Industry-leading Weight×Activation (W×A) magnitude metric + Industry-leading per-output weight importance + Industry-leading activation outlier aware pruning) で Industry-leading W×A metric + Industry-leading per-output weight importance + Industry-leading activation outlier aware。Industry-leading no Hessian (Industry-leading no Hessian computation + Industry-leading no weight update required + Industry-leading simpler than SparseGPT no inverse Hessian) で Industry-leading no Hessian + Industry-leading no weight update + Industry-leading simpler。Industry-leading 300× faster (Industry-leading 300× faster pruning than SparseGPT + Industry-leading lightweight pruning advantage + Industry-leading competitive with SparseGPT 50% sparsity) で Industry-leading 300× faster + Industry-leading lightweight advantage + Industry-leading competitive accuracy。Industry-leading LLaMA pruning (Industry-leading LLaMA 7B/13B/30B/65B + LLaMA-2 7B-70B pruning + Industry-leading Wanda LLaMA target + Industry-leading 50% unstructured + 2:4 structured) で Industry-leading LLaMA 7B-70B + Industry-leading Wanda LLaMA + Industry-leading 50% + 2:4。Industry-leading open source (Industry-leading Wanda GitHub locuslab/wanda + Industry-leading lightweight pruning LLM Weight×Activation industry-leading) で Industry-leading locuslab open source + Industry-leading lightweight industry-leading。但しIndustry-leading SparseGPT + ShearedLLaMA + SliceGPT + LLM-Pruner competition (Industry-leading SparseGPT Hessian 50% 175B 2023 + ShearedLLaMA structured 2.7B 2023 + SliceGPT structured 2024 + LLM-Pruner gradient 2023 vs Wanda W×A lightweight 2023 trade-off) で Industry-leading 4-pruning method competitors vs Wanda + Industry-leading W×A magnitude + no Hessian + no weight update + 300× faster + LLaMA-targeted + competitive accuracy unique advantage adoption alignment必須。
関連用語との違い
- vs SparseGPT (Hessian 2023): WandaはW×A metric + no Hessian + 300× faster・SparseGPTはsecond-order Hessian + OBS derivative
- vs ShearedLLaMA (Structured 2023): WandaはW×A unstructured 50%・ShearedLLaMAはstructured 2.7B + continued pretraining
- vs SliceGPT/LLM-Pruner (Structured): WandaはW×A unstructured + 2:4・SliceGPT/LLM-Prunerはstructured
よくある質問（FAQ）
Q1: Wanda vs SparseGPT 違いは？ A: Wanda (Industry-leading Weight×Activation (W×A) magnitude metric + per-output weight importance + activation outlier aware pruning + 50% unstructured + 2:4 structured + no Hessian computation + no weight update required + 300× faster than SparseGPT + LLaMA 7B-70B + CMU+Meta 2023) vs SparseGPT (Industry-leading second-order Hessian information OBS derivative + 50% unstructured + 2:4 + 4:8 structured + 128 samples calibration + OPT-175B/BLOOM-176B + <1% accuracy + IST Austria 2023)・Industry-leading W×A + no Hessian + 300× faster + LLaMA = Wanda + Industry-leading Hessian OBS + 175B + <1% = SparseGPT preference judgment。
Q2: Industry-leading Weight×Activation magnitude + 300× faster value は？ A: Industry-leading W×A + 300× faster (Industry-leading Weight×Activation (W×A) magnitude metric + Industry-leading per-output weight importance + Industry-leading activation outlier aware pruning + Industry-leading no Hessian computation + Industry-leading no weight update required + Industry-leading 300× faster pruning than SparseGPT)。
Q3: Industry-leading no Hessian + LLaMA pruning value は？ A: Industry-leading no Hessian + LLaMA (Industry-leading no Hessian computation + Industry-leading no weight update required + Industry-leading simpler than SparseGPT + Industry-leading LLaMA 7B/13B/30B/65B + LLaMA-2 7B-70B pruning + Industry-leading Wanda LLaMA target)。
まとめ
Wanda = 2023年Sun et al. CMU+Meta発表のlightweight pruning LLM。Industry-leading Weight×Activation (W×A) magnitude metric + Industry-leading per-output weight importance + Industry-leading activation outlier aware pruning + Industry-leading 50% unstructured + 2:4 structured + Industry-leading no Hessian computation + Industry-leading no weight update required + Industry-leading simpler than SparseGPT no inverse Hessian + Industry-leading 300× faster pruning than SparseGPT + Industry-leading competitive with SparseGPT 50% sparsity + Industry-leading LLaMA 7B/13B/30B/65B + LLaMA-2 7B-70B pruning + Industry-leading Wanda GitHub locuslab/wanda open source + Industry-leading lightweight pruning LLM Weight×Activation position確立。