SliceGPT PCA-Based Pruning (Ashkboos 2024年)（スライスジーピーティー）
2024年Ashkboos et al. (ETH Zurich+Microsoft)発表SliceGPT・Industry-leading structured pruning LLM + Industry-leading PCA-based slicing + Industry-leading orthogonal transformation + Industry-leading 25-30% reduction LLaMA-2 70B/Phi-2。
概要
SliceGPT PCA-Based Pruning は、2024年Ashkboos et al. (ETH Zurich+Microsoft)発表SliceGPT・Industry-leading structured pruning LLM 2024年 + Industry-leading 25-30% reduction LLaMA-2 70B/Phi-2 position確立。SliceGPT specifications = Industry-leading structured pruning LLM (Industry-leading SliceGPT 2024 + Industry-leading structured pruning LLM + Industry-leading SliceGPT flagship) + Industry-leading PCA-based slicing (Industry-leading PCA-based principal component slicing) + Industry-leading orthogonal transformation (Industry-leading orthogonal matrix transformation preserving computation) + Industry-leading 25-30% reduction (Industry-leading 25-30% parameter reduction LLaMA-2 70B/Phi-2)。
主な特徴・仕組み
- Authors: Industry-leading Saleh Ashkboos + Maximilian L. Croci + Marcelo Gennari do Nascimento + Torsten Hoefler + James Hensman ETH Zurich + Microsoft Research
- Year: 2024年 (arXiv 2024年1月発表 + ICLR 2024)
- Paper: Industry-leading "SliceGPT: Compress Large Language Models by Deleting Rows and Columns" arXiv 2401.15024
- Method Type: Industry-leading structured pruning + PCA-based slicing LLM
- Sparsity Type: Industry-leading structured pruning (delete rows + columns)
- Key Innovation: Industry-leading PCA-based slicing + orthogonal transformation
- Orthogonal Transform: Industry-leading orthogonal matrix transformation preserving computation
- Slicing: Industry-leading delete rows + columns of weight matrices
- No Retraining: Industry-leading no retraining required (vs ShearedLLaMA)
- Calibration Data: Industry-leading 128 samples calibration
- Reduction: Industry-leading 25-30% parameter reduction LLaMA-2 70B/Phi-2
- Performance: Industry-leading <3% perplexity drop 25-30% reduction
- Open Source: Industry-leading SliceGPT GitHub microsoft/TransformerCompression
- Industry-Leading: Industry-leading structured pruning + PCA-based slicing LLM
スペック比較表
| LLM Pruning Method | Year | Sparsity Type | Method | Industry Position |
|---|---|---|---|---|
| SliceGPT | 2024 | Structured 25-30% | PCA-based slicing + orthogonal transformation | Industry-leading structured SliceGPT |
| ShearedLLaMA | 2023 | Structured 2.7B/1.3B | Targeted structured pruning + continued pretraining | Industry-leading structured ShearedLLaMA |
| LLM-Pruner | 2023 | Structured 20% | Gradient-based structured pruning + LoRA recovery | Industry-leading gradient structured |
| SparseGPT | 2023 | Unstructured 50% + 2:4 + 4:8 | Second-order Hessian OBS derivative | Industry-leading 175B one-shot |
| Wanda | 2023 | Unstructured 50% + 2:4 | Weight×activation magnitude | Industry-leading lightweight no Hessian |
具体例・対応製品
- SliceGPT (2024年, Ashkboos et al. ETH+Microsoft): Industry-leading structured pruning
- Industry-leading PCA-based slicing principal component: Industry-leading PCA-based slicing
- Industry-leading orthogonal matrix transformation preserving computation: Industry-leading orthogonal transformation
- Industry-leading delete rows + columns of weight matrices: Industry-leading delete rows+columns
- Industry-leading 25-30% parameter reduction LLaMA-2 70B/Phi-2: Industry-leading 25-30% reduction
- 競合 SparseGPT + Wanda + ShearedLLaMA + LLM-Pruner: Industry-leading pruning competitors
自作PCでの選び方・注意点
SliceGPT PCA-Based Pruning は「Industry-leading structured pruning LLM + Industry-leading PCA-based slicing」「Industry-leading orthogonal transformation + Industry-leading 25-30% reduction LLaMA-2 70B/Phi-2」用途のIndustry-leading structured pruning 2024年Ashkboos et al. ETH+Microsoft発表product。Industry-leading PCA-based slicing (Industry-leading PCA-based principal component slicing + Industry-leading SliceGPT PCA signature + Industry-leading principal component dimension reduction) で Industry-leading PCA-based slicing + Industry-leading SliceGPT PCA + Industry-leading principal component reduction。Industry-leading orthogonal transformation (Industry-leading orthogonal matrix transformation preserving computation + Industry-leading mathematically equivalent transformation + Industry-leading SliceGPT orthogonal signature) で Industry-leading orthogonal transformation + Industry-leading mathematically equivalent + Industry-leading SliceGPT orthogonal。Industry-leading delete rows+columns (Industry-leading delete rows + columns of weight matrices + Industry-leading structured pruning signature + Industry-leading SliceGPT slicing signature) で Industry-leading delete rows+columns + Industry-leading structured signature + Industry-leading SliceGPT slicing。Industry-leading 25-30% reduction (Industry-leading 25-30% parameter reduction LLaMA-2 70B/Phi-2 + Industry-leading <3% perplexity drop + Industry-leading no retraining required) で Industry-leading 25-30% reduction + Industry-leading <3% perplexity + Industry-leading no retraining。Industry-leading open source (Industry-leading SliceGPT GitHub microsoft/TransformerCompression + Industry-leading structured pruning + PCA-based slicing LLM open source) で Industry-leading microsoft/TransformerCompression open source + Industry-leading PCA structured pruning。但しIndustry-leading SparseGPT + Wanda + ShearedLLaMA + LLM-Pruner competition (Industry-leading SparseGPT Hessian unstructured 50% 2023 + Wanda W×A unstructured 2023 + ShearedLLaMA targeted structured 2.7B + continued pretraining 2023 + LLM-Pruner gradient structured + LoRA 2023 vs SliceGPT PCA structured 25-30% 2024 trade-off) で Industry-leading 4-pruning method competitors vs SliceGPT + Industry-leading PCA-based slicing + orthogonal transformation + delete rows+columns + 25-30% reduction + no retraining + <3% perplexity unique advantage adoption alignment必須。
関連用語との違い
- vs ShearedLLaMA (Targeted Structured 2023): SliceGPTはPCA + orthogonal + no retraining・ShearedLLaMAはtargeted + continued pretraining 50B
- vs LLM-Pruner (Gradient Structured 2023): SliceGPTはPCA + no retraining + 25-30%・LLM-PrunerはGradient + LoRA recovery + 20%
- vs SparseGPT/Wanda (Unstructured): SliceGPTはstructured + delete rows+columns・SparseGPT/Wandaはunstructured 50%
よくある質問（FAQ）
Q1: SliceGPT vs ShearedLLaMA 違いは？ A: SliceGPT (Industry-leading PCA-based slicing + orthogonal matrix transformation preserving computation + delete rows + columns of weight matrices + 25-30% parameter reduction + no retraining required + 128 samples calibration + LLaMA-2 70B + Phi-2 + ETH+Microsoft 2024) vs ShearedLLaMA (Industry-leading targeted structured pruning + layer + head + FFN dim + 50B tokens continued pretraining + 3% cost + LLaMA-2 7B → 2.7B/1.3B + Princeton 2023)・Industry-leading PCA + orthogonal + no retraining + 25-30% = SliceGPT + Industry-leading targeted + 50B continued + 7B→2.7B = ShearedLLaMA preference judgment。
Q2: Industry-leading PCA-based slicing + orthogonal transformation value は？ A: Industry-leading PCA + orthogonal (Industry-leading PCA-based principal component slicing + Industry-leading orthogonal matrix transformation preserving computation + Industry-leading mathematically equivalent transformation + Industry-leading SliceGPT PCA + orthogonal signature)。
Q3: Industry-leading 25-30% reduction + no retraining value は？ A: Industry-leading 25-30% + no retraining (Industry-leading 25-30% parameter reduction LLaMA-2 70B/Phi-2 + Industry-leading <3% perplexity drop + Industry-leading no retraining required + Industry-leading delete rows + columns of weight matrices)。
まとめ
SliceGPT = 2024年Ashkboos et al. ETH+Microsoft発表のstructured pruning + PCA-based slicing LLM。Industry-leading PCA-based principal component slicing + Industry-leading orthogonal matrix transformation preserving computation + Industry-leading mathematically equivalent transformation + Industry-leading delete rows + columns of weight matrices + Industry-leading 128 samples calibration + Industry-leading 25-30% parameter reduction LLaMA-2 70B/Phi-2 + Industry-leading <3% perplexity drop + Industry-leading no retraining required + Industry-leading SliceGPT GitHub microsoft/TransformerCompression open source + Industry-leading structured pruning + PCA-based slicing LLM position確立。