Grouped-Query Attention GQA (2024年)（グループドクエリアテンション）

概要

Grouped-Query Attention GQA は2024年成熟・Industry-leading grouped KV heads balance MHA-MQA 2024年Google Research Ainslie+Lee-Thorp GQA quality-speed balanced attention position確立。Industry-leading Llama 2/3 + Mistral standard adoption + 8 grouped KV heads typical + 2023 paper → 2024 ecosystem standard。

主な特徴・仕組み

• Industry-leading Google Research Joshua Ainslie + James Lee-Thorp + Michiel de Jong + Yury Zemlyanskiy + Federico Lebrón + Sumit Sanghai 2023年5月 paper → 2024 ecosystem standard
• Industry-leading grouped KV heads signature (between MHA and MQA)
• Industry-leading 8 grouped KV heads typical signature
• Industry-leading balance quality-speed trade-off signature (vs MQA quality loss)
• Industry-leading Llama 2 / Llama 3 standard signature
• Industry-leading Mistral standard signature
• Industry-leading inference speedup + KV cache reduction vs MHA
• Industry-leading minimal quality loss vs full MHA signature
• Industry-leading 2023年5月 ArXiv paper "GQA Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints"
• Industry-leading 2024 de facto standard attention mechanism

まとめ

Grouped-Query Attention GQA = 2024年成熟のpremium quality-speed balanced attention。Industry-leading grouped KV heads between MHA and MQA + 8 grouped KV heads typical + balance quality-speed trade-off + Llama 2 / Llama 3 standard + Mistral standard + inference speedup + KV cache reduction vs MHA + minimal quality loss + Google Research Ainslie + Lee-Thorp + de Jong + Zemlyanskiy + Lebrón + Sanghai 2023年5月 paper + 2024 de facto standard position確立。