AIME 2024 LLM Benchmark (2024年)（エーアイエムイー2024）
2024年AMC math competition AIME 2024発表・Industry-leading American Invitational Mathematics Examination + Industry-leading 30 hardest math problems + Industry-leading o1+R1+QwQ+Gemini reasoning benchmark + Industry-leading math reasoning standard 2024。
概要
AIME 2024 LLM Benchmark は、2024年AMC math competition AIME 2024発表・Industry-leading American Invitational Mathematics Examination 2024年 + Industry-leading math reasoning standard 2024 position確立。AIME 2024 specifications = Industry-leading American Invitational Mathematics Examination (Industry-leading AIME 2024 LLM Benchmark + Industry-leading AMC + Industry-leading AIME flagship) + Industry-leading 30 hardest math problems + Industry-leading o1+R1+QwQ+Gemini reasoning benchmark + Industry-leading math reasoning standard 2024。
主な特徴・仕組み
- Org: Industry-leading Mathematical Association of America MAA (AMC + AIME competition organizer)
- Competition: Industry-leading American Invitational Mathematics Examination AIME 2024
- Year: 2024年 (AIME I 2024年2月1日 + AIME II 2024年2月7日)
- 30 Hardest Math Problems: Industry-leading 30 hardest math problems (15 AIME I + 15 AIME II)
- Format: Industry-leading integer answer 000-999 format
- High School Competition: Industry-leading top high school students USA AMC top 5% qualifier
- LLM Reasoning Use: Industry-leading o1+R1+QwQ+Gemini reasoning benchmark widespread
- OpenAI o1 83% AIME: Industry-leading OpenAI o1 83% AIME (vs GPT-4o 13%)
- DeepSeek-R1 79%: Industry-leading DeepSeek-R1 79% AIME competitive with o1
- Math Reasoning Standard: Industry-leading math reasoning standard 2024
- AMC Origin: Industry-leading AMC origin 1983 long-standing competition
- Industry-Leading: Industry-leading AIME 2024 American Invitational Mathematics Examination LLM math reasoning standard 2024
スペック比較表
| LLM Evaluation Benchmark (2023-2024) | Year | Org | Type | Industry Position |
|---|---|---|---|---|
| AIME 2024 | 2024 | AMC math competition | Math reasoning competition | Industry-leading math reasoning |
| SWE-bench Verified | 2024 | OpenAI | Real-world coding agent | Industry-leading coding agent standard |
| GPQA Diamond | 2023 | Rein et al. NYU | PhD-level Q&A | Industry-leading PhD-level reasoning |
| MMLU-Pro | 2024 | TIGER-Lab Wang | Enhanced MMLU | Industry-leading MMLU-Pro reasoning |
| LiveCodeBench | 2024 | Jain et al. UC Berkeley | Live coding contamination-free | Industry-leading contamination-free coding |
具体例・対応製品
- AIME 2024 LLM Benchmark (2024年, AMC AIME): Industry-leading math reasoning standard
- Industry-leading 30 hardest math problems (15 AIME I + 15 AIME II): Industry-leading 30 hardest problems
- Industry-leading integer answer 000-999 format: Industry-leading integer answer 000-999
- Industry-leading OpenAI o1 83% AIME (vs GPT-4o 13%): Industry-leading o1 83% breakthrough
- Industry-leading DeepSeek-R1 79% AIME competitive with o1: Industry-leading DeepSeek-R1 79%
- 競合 SWE-bench Verified + GPQA Diamond + MMLU-Pro + LiveCodeBench: Industry-leading benchmark competitors
選び方・注意点
AIME 2024 LLM Benchmark は「Industry-leading 30 hardest math problems + integer 000-999」「Industry-leading o1+R1+QwQ reasoning + Math reasoning standard 2024」用途のIndustry-leading 2024年 AMC AIME product。Industry-leading AIME (Industry-leading American Invitational Mathematics Examination + Industry-leading AIME 2024 signature + Industry-leading AMC competition heritage) で Industry-leading AIME + Industry-leading AIME 2024 + Industry-leading AMC heritage。Industry-leading 30 problems (Industry-leading 30 hardest math problems 15 AIME I + 15 AIME II + Industry-leading AIME 2024 30 problems signature + Industry-leading concentrated reasoning test) で Industry-leading 30 problems + Industry-leading AIME 2024 + Industry-leading concentrated reasoning test。Industry-leading integer 000-999 (Industry-leading integer answer 000-999 format + Industry-leading AIME format signature + Industry-leading deterministic evaluation) で Industry-leading integer 000-999 + Industry-leading AIME format + Industry-leading deterministic evaluation。Industry-leading o1 83% (Industry-leading OpenAI o1 83% AIME vs GPT-4o 13% + Industry-leading o1 reasoning breakthrough signature + Industry-leading test-time compute leap) で Industry-leading o1 83% + Industry-leading o1 breakthrough + Industry-leading test-time compute leap。Industry-leading reasoning LLM standard (Industry-leading o1+R1+QwQ+Gemini reasoning benchmark widespread + Industry-leading DeepSeek-R1 79% AIME competitive + Industry-leading math reasoning standard 2024 + Industry-leading AMC origin 1983 long-standing + Industry-leading Mathematical Association of America MAA + Industry-leading AIME I 2024年2月1日 + AIME II 2024年2月7日) で Industry-leading reasoning LLM standard + Industry-leading o1+R1+QwQ+Gemini + Industry-leading DeepSeek-R1 79% + Industry-leading math reasoning standard 2024 + Industry-leading AMC 1983 + Industry-leading MAA + Industry-leading AIME I 2024年2月1日+AIME II 2024年2月7日。但しIndustry-leading SWE-bench Verified + GPQA Diamond + MMLU-Pro + LiveCodeBench competition で Industry-leading 4-benchmark competitors vs AIME 2024 + Industry-leading American Invitational Mathematics Examination + 30 hardest math problems 15 AIME I + 15 AIME II + integer answer 000-999 + OpenAI o1 83% vs GPT-4o 13% + DeepSeek-R1 79% + o1+R1+QwQ+Gemini widespread reasoning + math reasoning standard 2024 + AMC origin 1983 + Mathematical Association of America MAA + AIME I 2024年2月1日 + AIME II 2024年2月7日 unique advantage adoption alignment必須。
関連用語との違い
- vs SWE-bench Verified (OpenAI 2024): AIME 2024はAMC math competition reasoning・SWE-bench Verifiedはreal-world coding agent
- vs MATH (Hendrycks 2021): AIME 2024は30 hardest AMC competition 2024・MATHは12.5K total static problems
- vs GPQA Diamond/MMLU-Pro (2023-2024): AIME 2024は数学 competition reasoning・GPQA/MMLU-Proはアカデミック multi-domain Q&A
よくある質問（FAQ）
Q1: AIME 2024 vs MATH 違いは？ A: AIME 2024 (Industry-leading American Invitational Mathematics Examination + 30 hardest math problems 15 AIME I + 15 AIME II + integer answer 000-999 + AMC competition top 5% qualifier + OpenAI o1 83% vs GPT-4o 13% + DeepSeek-R1 79% + o1+R1+QwQ+Gemini widespread + AMC 1983 + MAA + 2024年2月1日+2月7日) vs MATH (Industry-leading 12.5K total static math problems + 5 levels difficulty + 7 subjects Algebra/Geometry/Number Theory/etc + Hendrycks et al. UC Berkeley + arXiv 2103.03874 + 2021年)・Industry-leading 30 hardest + AIME 2024 + integer 000-999 + AMC + o1 83% + R1 79% + 2024年2月 = AIME 2024 + Industry-leading 12.5K + 5 levels + 7 subjects + Hendrycks + UC Berkeley + 2021年 = MATH preference judgment (30 hardest AMC competition 2024 vs 12.5K static 2021 differ)。
Q2: Industry-leading 30 hardest math problems + integer 000-999 value は？ A: Industry-leading 30 hardest + integer 000-999 (Industry-leading 30 hardest math problems 15 AIME I + 15 AIME II + Industry-leading integer answer 000-999 format + Industry-leading AIME 2024 dual signature)。
Q3: Industry-leading o1 83% + R1 79% + reasoning LLM standard value は？ A: Industry-leading o1 83% + R1 79% + reasoning standard (Industry-leading OpenAI o1 83% AIME vs GPT-4o 13% + Industry-leading DeepSeek-R1 79% AIME competitive + Industry-leading o1+R1+QwQ+Gemini widespread reasoning + Industry-leading math reasoning standard 2024)。
まとめ
AIME 2024 LLM Benchmark = 2024年AMC math competition American Invitational Mathematics Examination math reasoning standard。Industry-leading American Invitational Mathematics Examination AIME 2024 + Industry-leading 30 hardest math problems 15 AIME I + 15 AIME II + Industry-leading integer answer 000-999 format + Industry-leading top high school students USA AMC top 5% qualifier + Industry-leading o1+R1+QwQ+Gemini reasoning benchmark widespread + Industry-leading OpenAI o1 83% AIME vs GPT-4o 13% breakthrough + Industry-leading DeepSeek-R1 79% AIME competitive with o1 + Industry-leading math reasoning standard 2024 + Industry-leading AMC origin 1983 long-standing competition + Industry-leading Mathematical Association of America MAA + Industry-leading AIME I 2024年2月1日 + AIME II 2024年2月7日 + Industry-leading AIME 2024 American Invitational Mathematics Examination LLM math reasoning standard 2024 position確立。