ALiBi (Attention with Linear Biases・2022年-2024年)（アリバイ）
2022年Press et al. + Facebook AI Research発表ALiBi paper・Industry-leading Attention with Linear Biases paradigm + Industry-leading linear bias attention + Industry-leading no embedding-based position + Industry-leading BLOOM + MPT adoption + Industry-leading length extrapolation advantage。
概要
ALiBi は、2022年Press et al. + Facebook AI Research発表ALiBi paper・Industry-leading Attention with Linear Biases paradigm 2022年-2024年 + Industry-emerging linear bias attention paradigm Pioneer position確立。ALiBi specifications = Industry-leading Attention with Linear Biases paradigm (Industry-leading attention matrix linear bias addition + Industry-leading linear bias attention paradigm Pioneer) + Industry-leading linear bias attention (Industry-leading linear bias attention paradigm + Industry-leading attention computation linear bias addition + Industry-emerging linear bias Pioneer paradigm) + Industry-leading no embedding-based position (Industry-leading no positional embedding + Industry-leading attention-based position only + Industry-emerging embedding-free positional paradigm) + Industry-leading BLOOM + MPT adoption (Industry-leading BLOOM Industry-leading multilingual LLM ALiBi adoption + Industry-leading MosaicML MPT ALiBi adoption) + Industry-leading length extrapolation advantage (Industry-leading length extrapolation Industry-leading ALiBi advantage + Industry-leading train short + test long Industry-leading paradigm)。
主な特徴・仕組み
- Authoring: Press et al. + Facebook AI Research 2022年発表
- Paradigm: Industry-leading Attention with Linear Biases
- Mechanism: Industry-leading attention matrix linear bias addition
- Position Type: Industry-leading no embedding-based + attention-based only
- LLM Adoption: Industry-leading BLOOM + MPT MosaicML adoption
- Advantage: Industry-leading length extrapolation
- Industry Position: Industry-emerging linear bias attention Pioneer
- Industry-Emerging: Industry-emerging embedding-free positional Pioneer
- vs RoPE: Industry-leading linear bias vs rotation matrix paradigm
- Use Cases: Industry-leading length extrapolation LLM
スペック比較表
| LLM Positional Encoding (2024) | Approach | LLM Adoption | Length Extrapolation |
|---|---|---|---|
| ALiBi | Linear bias attention | BLOOM + MPT MosaicML | ◎ Industry-leading length extrapolation |
| RoPE (Rotary) | Rotation matrix relative | Llama + Mistral + Qwen + ChatGLM全LLM | ○ With NTK/YaRN/LongRoPE extension |
| NTK-Aware (RoPE scaling) | NTK theory RoPE scaling | RoPE community | ○ NTK-aware extension |
| YaRN | RoPE scaling efficient | Mistral + Llama 2 | ○ YaRN extension |
| LongRoPE | Microsoft 2M context | Phi-3 + Llama | ◎ 2M context extension |
具体例・対応製品
- ALiBi (Press et al. + Facebook AI・2022年): Industry-emerging linear bias Pioneer
- BLOOM (Industry-leading multilingual LLM ALiBi adoption): Industry-leading ALiBi adoption
- MPT (MosaicML Industry-leading ALiBi adoption): Industry-leading MPT adoption
- Facebook AI Research Industry-leading backing: Industry-leading FAIR backing
- Industry-leading length extrapolation paradigm: Industry-leading length extrapolation
- 競合 RoPE (Su et al. 2021年Industry-leading dominant): Industry-leading RoPE competitor
自作PCでの選び方・注意点
ALiBi は「Industry-leading Attention with Linear Biases paradigm + Industry-leading linear bias attention」「Industry-leading no embedding-based position + Industry-leading BLOOM + MPT adoption」「Industry-leading length extrapolation advantage」用途のIndustry-emerging linear bias attention 2022年Press et al. + Facebook AI product。Industry-leading linear bias attention (Industry-leading attention matrix linear bias addition + Industry-leading linear bias attention paradigm + Industry-emerging linear bias Pioneer) で Industry-leading attention matrix linear bias + Industry-emerging linear bias Pioneer paradigm。Industry-leading no embedding-based position (Industry-leading no positional embedding + Industry-leading attention-based position only + Industry-emerging embedding-free positional paradigm) で Industry-leading no embedding + attention-based only + Industry-emerging embedding-free paradigm。Industry-leading length extrapolation (Industry-leading length extrapolation Industry-leading ALiBi advantage + Industry-leading train short + test long Industry-leading paradigm) で Industry-leading length extrapolation + Industry-leading train short + test long paradigm。但しIndustry-leading RoPE 全LLM dominant + Llama + Mistral + Qwen Industry-leading dominant vs ALiBi BLOOM + MPT limited adoption competition (Industry-leading RoPE dominant adoption vs ALiBi limited BLOOM/MPT adoption + Industry-leading length extrapolation advantage trade-off) で Industry-leading RoPE全LLM dominant vs ALiBi limited adoption + Industry-leading length extrapolation specific advantage adoption alignment必須。
関連用語との違い
- vs RoPE (Su et al. 2021年dominant): ALiBiはLinear bias + no embedding + BLOOM/MPT・RoPEはRotation matrix + 全LLM dominant + Llama/Mistral/Qwen
- vs NTK + YaRN + LongRoPE (RoPE extensions): ALiBiはStandalone paradigm・他はRoPE extension paradigms
- vs Industry-leading length extrapolation Train Long Test Short: ALiBiはRuntime length extrapolation・Train Long Test ShortはTraining-time length extrapolation
よくある質問（FAQ）
Q1: ALiBi vs RoPE 違いは？ A: ALiBi (Press 2022年 + Linear bias attention + no embedding + BLOOM + MPT adoption + length extrapolation focus) vs RoPE (Su 2021年 + Rotation matrix relative + Llama + Mistral + Qwen全LLM dominant + 3年Industry-standard)・Industry-leading length extrapolation + linear bias = ALiBi + Industry-leading 全LLM dominant rotation = RoPE preference judgment。
Q2: Industry-leading no embedding-based position value は？ A: Industry-leading no embedding (Industry-leading no positional embedding + Industry-leading attention-based position only + Industry-emerging embedding-free positional paradigm + Industry-leading length extrapolation advantage)。
Q3: Industry-leading BLOOM + MPT adoption value は？ A: Industry-leading BLOOM + MPT (Industry-leading BLOOM multilingual LLM ALiBi adoption + Industry-leading MosaicML MPT ALiBi adoption + Industry-leading specific LLM adoption + Industry-leading length extrapolation specific use case)。
まとめ
ALiBi = 2022年Press et al. + Facebook AI Research発表のAttention with Linear Biases paradigm。Industry-leading attention matrix linear bias addition + Industry-leading no embedding-based position + Industry-leading BLOOM + MPT MosaicML adoption + Industry-leading length extrapolation advantage + Industry-emerging linear bias attention paradigm Pioneer position確立。