GPQA Diamond (2024年)（ジーピーキューエーダイヤモンド）

概要

GPQA Diamond は2024年成熟・Industry-leading 198 PhD-level questions 2024年NYU GPQA Diamond PhD-level reasoning benchmark position確立。Industry-leading expert 65% + non-expert 34% + Google-proof + biology/chemistry/physics + o1 78% + Claude 3.5 Sonnet 65% + 2024 reasoning benchmark gold standard。

主な特徴・仕組み

• Industry-leading NYU + Cohere + Anthropic (David Rein + Betty Li Hou et al.) 2023年11月 paper → 2024 standard
• Industry-leading GPQA Diamond hardest subset signature (198 questions)
• Industry-leading PhD-level questions signature (biology + chemistry + physics)
• Industry-leading expert 65% accuracy (PhD domain holders)
• Industry-leading non-expert 34% accuracy (PhD but different domain)
• Industry-leading Google-proof signature (questions resist web search)
• Industry-leading o1 78% (state-of-art 2024年9月)
• Industry-leading Claude 3.5 Sonnet 65%
• Industry-leading GPT-4o 53%
• Industry-leading 2024 reasoning benchmark gold standard signature
• Industry-leading 2023年11月 ArXiv paper "GPQA A Graduate-Level Google-Proof Q&A Benchmark"

まとめ

GPQA Diamond = 2024年成熟のpremium PhD-level reasoning benchmark。Industry-leading 198 PhD-level hardest subset questions biology + chemistry + physics + expert 65% non-expert 34% + Google-proof questions resist web search + o1 78% Claude 3.5 Sonnet 65% GPT-4o 53% + 2024 reasoning benchmark gold standard + NYU + Cohere + Anthropic David Rein + Betty Li Hou 2023年11月 paper position確立。