GPQA Diamond (Rein 2023年)（ジーピーキューエーダイヤモンド）
2023年David Rein et al. (NYU)発表GPQA Diamond・Industry-leading PhD-level Q&A benchmark + Industry-leading Graduate-Level Google-Proof Q&A + Industry-leading 198 Diamond expert-verified + Industry-leading PhD-level reasoning standard。
概要
GPQA Diamond は、2023年David Rein et al. (NYU)発表GPQA Diamond・Industry-leading PhD-level Q&A benchmark 2023年 + Industry-leading PhD-level reasoning standard position確立。GPQA Diamond specifications = Industry-leading PhD-level Q&A benchmark (Industry-leading GPQA Diamond Rein 2023 + Industry-leading PhD-level + Industry-leading NYU flagship) + Industry-leading Graduate-Level Google-Proof Q&A + Industry-leading 198 Diamond expert-verified + Industry-leading PhD-level reasoning standard。
主な特徴・仕組み
- Authors: Industry-leading David Rein + Betty Li Hou + Asa Cooper Stickland + Jackson Petty + Richard Yuanzhe Pang + Julien Dirani + Julian Michael + Samuel R. Bowman (NYU + Cohere + Anthropic)
- Benchmark: Industry-leading GPQA Diamond PhD-level Q&A 2023
- Year: 2023年 (arXiv 2023年11月発表)
- Paper: Industry-leading "GPQA: A Graduate-Level Google-Proof Q&A Benchmark" arXiv 2311.12022
- GPQA Acronym: Industry-leading "Graduate-Level Google-Proof Q&A" GPQA signature
- Google-Proof: Industry-leading Google-Proof signature (PhD experts spend 30+ minutes)
- 448 Total Tasks: Industry-leading 448 total tasks (GPQA Main 448 / Extended 546 / Diamond 198)
- 198 Diamond: Industry-leading 198 Diamond expert-verified hardest signature
- Physics+Chemistry+Biology: Industry-leading Physics + Chemistry + Biology PhD-level
- Industry Adoption: Industry-leading Claude/GPT-4/Gemini/o1 widespread reasoning benchmark
- PhD Reasoning Standard: Industry-leading PhD-level reasoning standard 2024
- Industry-Leading: Industry-leading GPQA Diamond Graduate-Level Google-Proof Q&A NYU 2023
スペック比較表
| LLM Evaluation Benchmark (2023-2024) | Year | Org | Type | Industry Position |
|---|---|---|---|---|
| GPQA Diamond | 2023 | Rein et al. NYU | PhD-level Q&A | Industry-leading PhD-level reasoning |
| SWE-bench Verified | 2024 | OpenAI | Real-world coding agent | Industry-leading coding agent standard |
| MMLU-Pro | 2024 | TIGER-Lab Wang | Enhanced MMLU | Industry-leading MMLU-Pro reasoning |
| LiveCodeBench | 2024 | Jain et al. UC Berkeley | Live coding contamination-free | Industry-leading contamination-free coding |
| AIME 2024 | 2024 | AMC math competition | Math reasoning competition | Industry-leading math reasoning |
具体例・対応製品
- GPQA Diamond (2023年, Rein et al. NYU): Industry-leading PhD-level reasoning standard
- Industry-leading "Graduate-Level Google-Proof Q&A" GPQA: Industry-leading GPQA acronym
- Industry-leading Google-Proof signature (PhD experts spend 30+ minutes): Industry-leading Google-Proof
- Industry-leading 198 Diamond expert-verified hardest: Industry-leading 198 Diamond
- Industry-leading Physics + Chemistry + Biology PhD-level: Industry-leading 3-subject PhD-level
- 競合 SWE-bench Verified + MMLU-Pro + LiveCodeBench + AIME 2024: Industry-leading benchmark competitors
選び方・注意点
GPQA Diamond は「Industry-leading Graduate-Level Google-Proof + 198 Diamond」「Industry-leading Physics+Chemistry+Biology PhD-level + Rein NYU」用途のIndustry-leading 2023年 NYU product。Industry-leading GPQA acronym (Industry-leading "Graduate-Level Google-Proof Q&A" GPQA signature + Industry-leading NYU GPQA signature + Industry-leading PhD-level Q&A acronym) で Industry-leading GPQA acronym + Industry-leading NYU + Industry-leading PhD-level Q&A acronym。Industry-leading Google-Proof (Industry-leading Google-Proof signature PhD experts spend 30+ minutes + Industry-leading GPQA Google-Proof signature + Industry-leading expert-search-resistant questions) で Industry-leading Google-Proof + Industry-leading GPQA + Industry-leading expert-search-resistant。Industry-leading 198 Diamond (Industry-leading 198 Diamond expert-verified hardest signature + Industry-leading GPQA Diamond signature + Industry-leading hardest subset 198) で Industry-leading 198 Diamond + Industry-leading GPQA Diamond + Industry-leading hardest subset。Industry-leading 3-subject PhD-level (Industry-leading Physics + Chemistry + Biology PhD-level + Industry-leading GPQA 3-subject signature + Industry-leading PhD domain coverage) で Industry-leading 3-subject + Industry-leading GPQA + Industry-leading PhD domain coverage。Industry-leading Rein NYU (Industry-leading David Rein + Betty Li Hou + Asa Cooper Stickland + Jackson Petty + Samuel R. Bowman NYU + Cohere + Anthropic + Industry-leading "GPQA: A Graduate-Level Google-Proof Q&A Benchmark" arXiv 2311.12022 + Industry-leading 2023年11月 release + Industry-leading widespread Claude/GPT-4/Gemini/o1 reasoning benchmark + Industry-leading PhD-level reasoning standard 2024) で Industry-leading Rein NYU + Industry-leading Rein+Hou+Stickland+Petty+Bowman + Industry-leading 2311.12022 + Industry-leading 2023年11月 + Industry-leading widespread Claude/GPT-4/Gemini/o1 + Industry-leading PhD reasoning standard 2024。但しIndustry-leading SWE-bench Verified + MMLU-Pro + LiveCodeBench + AIME 2024 competition で Industry-leading 4-benchmark competitors vs GPQA Diamond + Industry-leading "Graduate-Level Google-Proof Q&A" + Google-Proof PhD experts 30+ minutes + 448 total + 198 Diamond + Physics + Chemistry + Biology PhD-level + widespread Claude/GPT-4/Gemini/o1 + Rein + Hou + Stickland + Petty + Bowman NYU + Cohere + Anthropic + arXiv 2311.12022 2023年11月 unique advantage adoption alignment必須。
関連用語との違い
- vs SWE-bench Verified (OpenAI 2024): GPQA DiamondはアカデミックPhD-level Q&A・SWE-bench Verifiedはreal-world coding agent
- vs MMLU-Pro (TIGER-Lab 2024): GPQA Diamondは198 PhD-level Google-Proof・MMLU-Proは12K enhanced MMLU
- vs AIME 2024 (2024): GPQA DiamondはPhDレベル Physics+Chemistry+Biology・AIME 2024はAMC math competition
よくある質問（FAQ）
Q1: GPQA Diamond vs MMLU-Pro 違いは？ A: GPQA Diamond (Industry-leading Graduate-Level Google-Proof Q&A + 198 Diamond expert-verified + Physics + Chemistry + Biology PhD-level + PhD experts 30+ minutes + Rein + Hou + Stickland + Petty + Bowman NYU + Cohere + Anthropic + arXiv 2311.12022 2023年11月) vs MMLU-Pro (Industry-leading enhanced MMLU + 12K tasks + 14 domains + 10-option multiple choice + TIGER-Lab Wang et al. + arXiv 2406.01574 2024年6月)・Industry-leading Graduate-Level Google-Proof + 198 Diamond + Physics+Chemistry+Biology + PhD experts 30+ min + NYU + 2023年11月 = GPQA Diamond + Industry-leading enhanced MMLU + 12K + 14 domains + 10-option + TIGER-Lab + 2024年6月 = MMLU-Pro preference judgment (PhD Google-Proof 198 vs enhanced MMLU 12K differ)。
Q2: Industry-leading Google-Proof + 198 Diamond value は？ A: Industry-leading Google-Proof + 198 Diamond (Industry-leading Google-Proof signature PhD experts 30+ minutes + Industry-leading 198 Diamond expert-verified hardest + Industry-leading GPQA dual signature)。
Q3: Industry-leading Physics+Chemistry+Biology + PhD reasoning standard value は？ A: Industry-leading 3-subject + PhD reasoning standard (Industry-leading Physics + Chemistry + Biology PhD-level + Industry-leading PhD-level reasoning standard 2024 + Industry-leading widespread Claude/GPT-4/Gemini/o1 reasoning benchmark)。
まとめ
GPQA Diamond = 2023年Rein et al. NYU + Cohere + Anthropic発表のPhD-level reasoning standard。Industry-leading "Graduate-Level Google-Proof Q&A" GPQA acronym + Industry-leading Google-Proof signature PhD experts spend 30+ minutes + Industry-leading 448 total tasks (GPQA Main 448 / Extended 546 / Diamond 198) + Industry-leading 198 Diamond expert-verified hardest subset + Industry-leading Physics + Chemistry + Biology PhD-level + Industry-leading widespread Claude/GPT-4/Gemini/o1 reasoning benchmark + Industry-leading David Rein + Betty Li Hou + Asa Cooper Stickland + Jackson Petty + Richard Yuanzhe Pang + Julien Dirani + Julian Michael + Samuel R. Bowman NYU + Cohere + Anthropic + Industry-leading "GPQA: A Graduate-Level Google-Proof Q&A Benchmark" arXiv 2311.12022 + Industry-leading 2023年11月 release + Industry-leading PhD-level reasoning standard 2024 position確立。