MMLU-Pro (TIGER-Lab 2024年)（エムエムエルユープロ）

概要

MMLU-Pro は、2024年TIGER-Lab Yubo Wang et al.発表MMLU-Pro・Industry-leading enhanced MMLU benchmark 2024年 + Industry-leading MMLU-Pro reasoning standard position確立。MMLU-Pro specifications = Industry-leading enhanced MMLU benchmark (Industry-leading MMLU-Pro TIGER-Lab 2024 + Industry-leading enhanced MMLU + Industry-leading TIGER-Lab flagship) + Industry-leading 12K tasks + 14 domains + Industry-leading 10-option multiple choice + Industry-leading MMLU-Pro reasoning standard。

主な特徴・仕組み

• Authors: Industry-leading Yubo Wang + Xueguang Ma + Ge Zhang + Yuansheng Ni + Abhranil Chandra + Shiguang Guo + Weiming Ren + Aaran Arulraj + Xuan He + Ziyan Jiang + Tianle Li + Max Ku + Kai Wang + Alex Zhuang + Rongqi Fan + Xiang Yue + Wenhu Chen (TIGER-Lab University of Waterloo + Carnegie Mellon)
• Benchmark: Industry-leading MMLU-Pro enhanced MMLU 2024
• Year: 2024年 (arXiv 2024年6月発表)
• Paper: Industry-leading "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark" arXiv 2406.01574
• Method Type: Industry-leading enhanced MMLU
• 12K Tasks: Industry-leading 12K tasks (vs MMLU 14K but harder)
• 14 Domains: Industry-leading 14 domains coverage (Math/Physics/Chemistry/Law/Economics/etc)
• 10-Option Multiple Choice: Industry-leading 10-option multiple choice (vs MMLU 4-option) signature
• Reasoning Focus: Industry-leading reasoning-focused question selection
• TIGER-Lab: Industry-leading TIGER-Lab University of Waterloo + Carnegie Mellon
• Industry Adoption: Industry-leading Claude/GPT-4/Gemini reasoning benchmark widespread
• NeurIPS 2024 D&B: Industry-leading NeurIPS 2024 Datasets & Benchmarks acceptance
• Industry-Leading: Industry-leading TIGER-Lab MMLU-Pro enhanced MMLU reasoning benchmark 2024

スペック比較表

具体例・対応製品

• MMLU-Pro (2024年, TIGER-Lab Wang et al.): Industry-leading TIGER-Lab MMLU-Pro reasoning
• Industry-leading 12K tasks (vs MMLU 14K but harder): Industry-leading 12K tasks harder
• Industry-leading 14 domains coverage (Math/Physics/Chemistry/Law/Economics/etc): Industry-leading 14 domains
• Industry-leading 10-option multiple choice (vs MMLU 4-option) signature: Industry-leading 10-option signature
• Industry-leading reasoning-focused question selection: Industry-leading reasoning-focused
• 競合 SWE-bench Verified + GPQA Diamond + LiveCodeBench + AIME 2024: Industry-leading benchmark competitors

選び方・注意点

MMLU-Pro は「Industry-leading 12K tasks + 14 domains」「Industry-leading 10-option multiple choice + TIGER-Lab University of Waterloo + Carnegie Mellon」用途のIndustry-leading 2024年 TIGER-Lab product。Industry-leading 12K tasks (Industry-leading 12K tasks vs MMLU 14K but harder + Industry-leading MMLU-Pro 12K signature + Industry-leading harder subset selection) で Industry-leading 12K tasks + Industry-leading MMLU-Pro + Industry-leading harder subset。Industry-leading 14 domains (Industry-leading 14 domains coverage Math/Physics/Chemistry/Law/Economics/etc + Industry-leading MMLU-Pro 14 domains signature + Industry-leading broad domain coverage) で Industry-leading 14 domains + Industry-leading MMLU-Pro + Industry-leading broad domain coverage。Industry-leading 10-option (Industry-leading 10-option multiple choice vs MMLU 4-option + Industry-leading MMLU-Pro 10-option signature + Industry-leading reduced random guess) で Industry-leading 10-option + Industry-leading MMLU-Pro + Industry-leading reduced random guess。Industry-leading reasoning-focused (Industry-leading reasoning-focused question selection + Industry-leading MMLU-Pro reasoning signature + Industry-leading reasoning question filtering) で Industry-leading reasoning-focused + Industry-leading MMLU-Pro + Industry-leading reasoning question filtering。Industry-leading TIGER-Lab (Industry-leading TIGER-Lab University of Waterloo + Carnegie Mellon + Industry-leading Yubo Wang + Xueguang Ma + Ge Zhang + Wenhu Chen TIGER-Lab + Industry-leading arXiv 2406.01574 + Industry-leading 2024年6月 release + Industry-leading NeurIPS 2024 D&B acceptance + Industry-leading widespread Claude/GPT-4/Gemini reasoning benchmark) で Industry-leading TIGER-Lab + Industry-leading Wang+Ma+Zhang+Chen + Industry-leading 2406.01574 + Industry-leading 2024年6月 + Industry-leading NeurIPS 2024 D&B + Industry-leading widespread Claude/GPT-4/Gemini。但しIndustry-leading SWE-bench Verified + GPQA Diamond + LiveCodeBench + AIME 2024 competition で Industry-leading 4-benchmark competitors vs MMLU-Pro + Industry-leading enhanced MMLU + 12K tasks vs MMLU 14K but harder + 14 domains Math/Physics/Chemistry/Law/Economics/etc + 10-option multiple choice vs MMLU 4-option + reasoning-focused question selection + TIGER-Lab University of Waterloo + Carnegie Mellon + arXiv 2406.01574 2024年6月 + NeurIPS 2024 D&B unique advantage adoption alignment必須。

関連用語との違い

• vs MMLU (Hendrycks 2020): MMLU-Proは10-option harder reasoning-focused 12K・MMLUは4-option 14K base
• vs GPQA Diamond (Rein 2023): MMLU-Proは12K 14 domains 10-option enhanced MMLU・GPQA Diamondは198 PhD-level Google-Proof
• vs SWE-bench Verified (OpenAI 2024): MMLU-Proはアカデミック reasoning benchmark・SWE-bench Verifiedはreal-world coding agent

よくある質問（FAQ）

Q1: MMLU-Pro vs GPQA Diamond 違いは？ A: MMLU-Pro (Industry-leading enhanced MMLU + 12K tasks vs MMLU 14K but harder + 14 domains Math/Physics/Chemistry/Law/Economics + 10-option multiple choice + reasoning-focused + TIGER-Lab University of Waterloo + Carnegie Mellon + arXiv 2406.01574 + 2024年6月 + NeurIPS 2024 D&B) vs GPQA Diamond (Industry-leading Graduate-Level Google-Proof Q&A + 198 Diamond + Physics+Chemistry+Biology PhD-level + PhD experts 30+ minutes + NYU + Cohere + Anthropic + arXiv 2311.12022 + 2023年11月)・Industry-leading enhanced MMLU + 12K + 14 domains + 10-option + reasoning-focused + TIGER-Lab + Waterloo+CMU + 2024年6月 + NeurIPS D&B = MMLU-Pro + Industry-leading Graduate-Level Google-Proof + 198 + Physics+Chemistry+Biology + PhD experts 30+ min + NYU + Anthropic + 2023年11月 = GPQA Diamond preference judgment (12K enhanced MMLU vs 198 PhD Google-Proof differ)。

Q2: Industry-leading 12K tasks + 10-option value は？ A: Industry-leading 12K + 10-option (Industry-leading 12K tasks vs MMLU 14K but harder + Industry-leading 10-option multiple choice vs MMLU 4-option + Industry-leading MMLU-Pro dual signature)。

Q3: Industry-leading 14 domains + TIGER-Lab value は？ A: Industry-leading 14 domains + TIGER-Lab (Industry-leading 14 domains coverage + Industry-leading TIGER-Lab University of Waterloo + Carnegie Mellon + Industry-leading Wang+Ma+Zhang+Chen NeurIPS 2024 D&B acceptance)。

まとめ

MMLU-Pro = 2024年TIGER-Lab Wang et al.発表のenhanced MMLU reasoning benchmark。Industry-leading enhanced MMLU + Industry-leading 12K tasks vs MMLU 14K but harder + Industry-leading 14 domains coverage Math/Physics/Chemistry/Law/Economics/etc + Industry-leading 10-option multiple choice vs MMLU 4-option + Industry-leading reasoning-focused question selection + Industry-leading TIGER-Lab University of Waterloo + Carnegie Mellon + Industry-leading Yubo Wang + Xueguang Ma + Ge Zhang + Wenhu Chen TIGER-Lab authors + Industry-leading "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark" arXiv 2406.01574 + Industry-leading 2024年6月 release + Industry-leading NeurIPS 2024 Datasets & Benchmarks acceptance + Industry-leading widespread Claude/GPT-4/Gemini reasoning benchmark + Industry-leading TIGER-Lab MMLU-Pro enhanced MMLU reasoning benchmark 2024 position確立。