AQLM Additive Quantization (Egiazarian 2024年)（エーキューエルエム）
2024年Egiazarian et al. (Yandex)発表AQLM・Industry-leading 2-bit additive quantization LLM + Industry-leading learned codebooks + Industry-leading vector quantization + Industry-leading LLaMA-2 70B 2-bit production via Yandex。
概要
AQLM Additive Quantization は、2024年Egiazarian et al. (Yandex)発表AQLM・Industry-leading 2-bit additive quantization LLM 2024年 + Industry-leading LLaMA-2 70B 2-bit production via Yandex position確立。AQLM specifications = Industry-leading 2-bit additive quantization LLM (Industry-leading AQLM 2024 + Industry-leading 2-bit additive quantization LLM + Industry-leading AQLM flagship) + Industry-leading learned codebooks (Industry-leading learned codebooks 2-bit + Industry-leading vector quantization codebook learning) + Industry-leading vector quantization (Industry-leading additive vector quantization AQ) + Industry-leading LLaMA-2 70B 2-bit production via Yandex (Industry-leading 70B 2-bit production Yandex)。
主な特徴・仕組み
- Authors: Industry-leading Vage Egiazarian + Andrei Panferov + Denis Kuznedelev + Elias Frantar + Artem Babenko + Dan Alistarh Yandex Research + IST Austria
- Year: 2024年 (arXiv 2024年2月発表)
- Paper: Industry-leading "Extreme Compression of Large Language Models via Additive Quantization" arXiv 2401.06118
- Quantization Type: Industry-leading 2-bit additive quantization
- Bits: Industry-leading 2-bit additive (8×memory reduction FP16→INT2)
- Method: Industry-leading additive vector quantization AQ + Industry-leading learned codebooks + Industry-leading group-wise quantization + Industry-leading finetuning codebook
- Key Insight: Industry-leading "additive quantization with learned codebooks outperforms uniform/lattice at 2-bit"
- Memory Reduction: Industry-leading 8×memory reduction FP16→INT2
- Accuracy: Industry-leading LLaMA-2 70B 2-bit competitive with 4-bit GPTQ/AWQ
- Codebook: Industry-leading learned codebooks per-group vector quantization
- Production Adoption: Industry-leading HuggingFace Transformers + Yandex production
- Industry-Leading: Industry-leading 2-bit additive quantization LLM extreme
スペック比較表
| LLM Quantization Method | Year | Bits | Method | Industry Position |
|---|---|---|---|---|
| AQLM | 2024 | 2-bit | Additive quantization with learned codebooks | Industry-leading 2-bit additive quantization |
| QuIP/QuIP# | 2024 | 2-bit | Incoherence processing + E8 lattice quantization | Industry-leading 2-bit extreme quantization |
| GPTQ | 2022 | 4/3/2-bit | Second-order Hessian PTQ weight-only | Industry-leading 4-bit PTQ 175B |
| AWQ | 2023 | 4-bit | Activation-aware weight quantization | Industry-leading activation-aware quantization |
| SmoothQuant | 2022 | INT8 | Smoothing activation+weight quantization | Industry-leading W8A8 production |
具体例・対応製品
- AQLM (2024年, Egiazarian et al. Yandex): Industry-leading 2-bit additive quantization
- Industry-leading 2-bit additive quantization: Industry-leading 2-bit additive
- Industry-leading learned codebooks vector quantization: Industry-leading learned codebooks
- Industry-leading additive vector quantization AQ: Industry-leading additive vector quantization
- Industry-leading LLaMA-2 70B 2-bit production via Yandex: Industry-leading 70B 2-bit Yandex production
- 競合 GPTQ + AWQ + SmoothQuant + QuIP: Industry-leading quantization competitors
自作PCでの選び方・注意点
AQLM Additive Quantization は「Industry-leading 2-bit additive quantization LLM + Industry-leading learned codebooks」「Industry-leading vector quantization + Industry-leading LLaMA-2 70B 2-bit production via Yandex」用途のIndustry-leading 2-bit additive quantization 2024年Egiazarian et al. Yandex発表product。Industry-leading 2-bit additive (Industry-leading 2-bit additive quantization + Industry-leading 8×memory reduction FP16→INT2 + Industry-leading additive vector quantization AQ) で Industry-leading 2-bit additive + Industry-leading 8×memory reduction + Industry-leading additive vector quantization。Industry-leading learned codebooks (Industry-leading learned codebooks 2-bit + Industry-leading vector quantization codebook learning + Industry-leading group-wise quantization + Industry-leading finetuning codebook + Industry-leading "additive quantization with learned codebooks outperforms uniform/lattice at 2-bit" key insight) で Industry-leading learned codebooks + Industry-leading vector quantization codebook + Industry-leading group-wise + Industry-leading finetuning + Industry-leading additive insight。Industry-leading LLaMA-2 70B 2-bit Yandex (Industry-leading LLaMA-2 70B 2-bit production via Yandex + Industry-leading 2-bit competitive with 4-bit GPTQ/AWQ + Industry-leading Yandex Research + IST Austria + Industry-leading HuggingFace Transformers ecosystem) で Industry-leading 70B 2-bit Yandex + Industry-leading 2-bit vs 4-bit GPTQ/AWQ + Industry-leading Yandex Research + Industry-leading HuggingFace Transformers。Industry-leading codebook learning (Industry-leading codebook learning per-group vector quantization + Industry-leading AQLM flagship + Industry-leading 2-bit additive quantization LLM extreme) で Industry-leading codebook learning + Industry-leading AQLM flagship + Industry-leading 2-bit extreme。但しIndustry-leading GPTQ + AWQ + SmoothQuant + QuIP competition (Industry-leading GPTQ 4-bit Hessian PTQ 2022 + AWQ 4-bit activation-aware 2023 + SmoothQuant W8A8 INT8 2022 + QuIP 2-bit lattice 2024 vs AQLM 2-bit additive 2024 trade-off) で Industry-leading 4-quantization method competitors vs AQLM + Industry-leading 2-bit additive + learned codebooks + vector quantization + 70B 2-bit Yandex + HuggingFace Transformers unique advantage adoption alignment必須。
関連用語との違い
- vs QuIP (2-bit lattice 2024): AQLMはadditive + learned codebooks・QuIPはincoherence + E8 lattice
- vs GPTQ/AWQ (4-bit 2022-2023): AQLMは2-bit additive extreme・GPTQ/AWQは4-bit weight-only PTQ
- vs SmoothQuant (W8A8 2022): AQLMは2-bit weight-only extreme・SmoothQuantはINT8 W8A8 production
よくある質問（FAQ）
Q1: AQLM vs QuIP 違いは？ A: AQLM (Industry-leading 2-bit additive quantization + learned codebooks + vector quantization + group-wise + finetuning codebook + LLaMA-2 70B 2-bit production via Yandex + HuggingFace Transformers ecosystem) vs QuIP (Industry-leading 2-bit weight-only extreme + incoherence processing random rotation + Hadamard transform + E8 lattice 8-dim codebook + LLaMA-2 70B 2-bit production + Cornell 2024)・Industry-leading 2-bit + additive + learned codebooks + Yandex = AQLM + Industry-leading 2-bit + incoherence + E8 lattice + Cornell = QuIP preference judgment。
Q2: Industry-leading learned codebooks + vector quantization value は？ A: Industry-leading learned codebooks (Industry-leading learned codebooks 2-bit + Industry-leading vector quantization codebook learning + Industry-leading group-wise quantization + Industry-leading finetuning codebook + Industry-leading "additive quantization with learned codebooks outperforms uniform/lattice at 2-bit" key insight)。
Q3: Industry-leading 2-bit additive quantization + LLaMA-2 70B Yandex value は？ A: Industry-leading 2-bit additive (Industry-leading 2-bit additive quantization + Industry-leading additive vector quantization AQ + Industry-leading 8×memory reduction FP16→INT2 + Industry-leading LLaMA-2 70B 2-bit production via Yandex + Industry-leading 2-bit competitive with 4-bit GPTQ/AWQ)。
まとめ
AQLM = 2024年Egiazarian et al. Yandex発表の2-bit additive quantization LLM。Industry-leading 2-bit additive quantization + Industry-leading learned codebooks 2-bit + Industry-leading vector quantization codebook learning + Industry-leading additive vector quantization AQ + Industry-leading group-wise quantization + Industry-leading finetuning codebook + Industry-leading "additive quantization with learned codebooks outperforms uniform/lattice at 2-bit" key insight + Industry-leading 8×memory reduction FP16→INT2 + Industry-leading LLaMA-2 70B 2-bit production via Yandex + Industry-leading 2-bit competitive with 4-bit GPTQ/AWQ + Industry-leading Yandex Research + IST Austria + Industry-leading HuggingFace Transformers + Yandex production ecosystem + Industry-leading 2-bit additive quantization LLM extreme position確立。