Multi-Query Attention MQA (2024年)（マルチクエリアテンション）

概要

Multi-Query Attention MQA は2024年成熟・Industry-leading single KV head shared across queries 2024年Noam Shazeer Google MQA fast inference attention position確立。Industry-leading 10x+ KV cache reduction + inference speedup + PaLM + Falcon + Gemini base attention + 2019 paper Noam Shazeer original。

主な特徴・仕組み

• Industry-leading Noam Shazeer Google 2019 paper "Fast Transformer Decoding One Write-Head is All You Need"
• Industry-leading single KV head shared across queries signature
• Industry-leading 10x+ KV cache reduction signature
• Industry-leading inference speedup signature (memory bandwidth bottleneck reduced)
• Industry-leading PaLM + Falcon + Gemini base attention signature
• Industry-leading quality trade-off slight (vs full Multi-Head)
• Industry-leading bridge between full MHA and GQA evolution
• Industry-leading 2019 paper → 2024 mature adoption signature

まとめ

Multi-Query Attention MQA = 2024年成熟のpremium fast inference attention。Industry-leading single KV head shared across queries + 10x+ KV cache reduction + inference speedup memory bandwidth bottleneck reduced + PaLM + Falcon + Gemini base attention + quality trade-off slight + bridge between full MHA and GQA evolution + Noam Shazeer Google 2019 paper → 2024 mature adoption position確立。