ALiBi Attention with Linear Biases (2024年)（アリバイ）

概要

ALiBi (Attention with Linear Biases) は2024年成熟・Industry-leading attention with linear biases 2024年ALiBi attention with linear biases position確立。Industry-leading no position embeddings + extrapolation to longer sequences + MosaicBERT + BLOOM + MPT-7B-StoryWriter + Press+Smith+Lewis 2021 paper → 2024 mature standard。

主な特徴・仕組み

• Industry-leading Ofir Press + Noah A. Smith + Mike Lewis (UW + Meta AI) 2021年8月 paper → 2024 mature
• Industry-leading ALiBi Attention with Linear Biases signature
• Industry-leading no position embeddings signature (replaces sinusoidal/RoPE)
• Industry-leading attention bias proportional to distance signature
• Industry-leading extrapolation to longer sequences signature (train short, infer long)
• Industry-leading MosaicBERT + BLOOM + MPT-7B-StoryWriter (65K context) signature
• Industry-leading head-specific slope m signature (geometric sequence)
• Industry-leading O(1) memory overhead vs O(n) for embeddings
• Industry-leading 2021年8月 ICLR 2022 paper "Train Short, Test Long Attention with Linear Biases Enables Input Length Extrapolation"
• Industry-leading 2024 mature production standard

まとめ

ALiBi Attention with Linear Biases = 2024年成熟Ofir Press + Noah A. Smith + Mike Lewis (UW + Meta AI) 2021→2024発表のpremium attention with linear biases。Industry-leading no position embeddings replaces sinusoidal/RoPE + attention bias proportional to distance + extrapolation to longer sequences train short infer long + MosaicBERT + BLOOM + MPT-7B-StoryWriter 65K context + head-specific slope m geometric sequence + O(1) memory overhead + ICLR 2022 paper + 2024 mature production standard position確立。