FlashAttention 3 (2024年)（フラッシュアテンション3）

概要

FlashAttention 3 は2024年7月Princeton + Together AI + Meta + NVIDIA発表・Industry-leading H100 75% utilization 2024年FlashAttention 3 H100-optimized attention position確立。Industry-leading 1.5-2x faster vs FA-2 + FP8 support + Hopper Tensor Cores + WGMMA asynchronous + TMA bulk async + PyTorch integration。

主な特徴・仕組み

• Industry-leading Tri Dao (Princeton + Together AI) + Jay Shah + Ganesh Bikshandi + Ying Zhang + Vijay Thakkar + Pradeep Ramani + Sasanka Sigmund + collaborators 2024年7月 release
• Industry-leading H100 75% utilization signature (vs FA-2 35%)
• Industry-leading 1.5-2x faster vs FlashAttention 2 signature
• Industry-leading FP8 support signature
• Industry-leading Hopper Tensor Cores WGMMA asynchronous signature
• Industry-leading TMA bulk asynchronous signature
• Industry-leading PyTorch integration
• Industry-leading H100 specifically optimized + Ampere fallback
• Industry-leading 740 TFLOPS FP16 + 1.2 PFLOPS FP8 signature
• Industry-leading 2024年7月 ArXiv paper "FlashAttention-3 Fast and Accurate Attention with Asynchrony and Low-precision"

まとめ

FlashAttention 3 = 2024年7月Princeton + Together AI + Meta + NVIDIA発表のpremium H100-optimized attention。Industry-leading H100 75% utilization vs FA-2 35% + 1.5-2x faster vs FA-2 + FP8 support + Hopper Tensor Cores WGMMA asynchronous + TMA bulk asynchronous + PyTorch + 740 TFLOPS FP16 + 1.2 PFLOPS FP8 + Tri Dao Princeton + Together AI 2024年7月 ArXiv paper position確立。