SWE-bench Verified (OpenAI 2024年)（スイーベンチベリファイド）
2024年OpenAI発表SWE-bench Verified・Industry-leading SWE-bench human-verified subset + Industry-leading 500 GitHub issue tasks + Industry-leading real-world coding benchmark + Industry-leading agent benchmark standard 2024。
概要
SWE-bench Verified は、2024年OpenAI発表SWE-bench Verified・Industry-leading SWE-bench human-verified subset 2024年 + Industry-leading agent benchmark standard 2024 position確立。SWE-bench Verified specifications = Industry-leading SWE-bench human-verified subset (Industry-leading SWE-bench Verified 2024 + Industry-leading human-verified + Industry-leading OpenAI flagship) + Industry-leading 500 GitHub issue tasks + Industry-leading real-world coding benchmark + Industry-leading agent benchmark standard 2024。
主な特徴・仕組み
- Org: Industry-leading OpenAI (Founded 2015 San Francisco)
- Benchmark: Industry-leading SWE-bench Verified human-verified subset 2024
- Year: 2024年 (SWE-bench Verified release 2024年8月)
- Base: Industry-leading SWE-bench Carlini et al. 2023 base (Princeton 2310.06770)
- Verified Subset: Industry-leading human-verified subset signature
- 500 Tasks: Industry-leading 500 GitHub issue tasks subset
- 12 Repositories: Industry-leading 12 Python repositories (Django/Sympy/etc)
- Industry Adoption: Industry-leading Claude/GPT/Devin/Cursor benchmark standard
- Coding Agent Standard: Industry-leading coding agent benchmark standard 2024
- 2024年8月: Industry-leading 2024年8月 OpenAI release
- Industry-Leading: Industry-leading SWE-bench Verified human-verified agent benchmark standard 2024
スペック比較表
| LLM Evaluation Benchmark (2023-2024) | Year | Org | Type | Industry Position |
|---|---|---|---|---|
| SWE-bench Verified | 2024 | OpenAI | Real-world coding agent | Industry-leading coding agent standard |
| GPQA Diamond | 2023 | Rein et al. NYU | PhD-level Q&A | Industry-leading PhD-level reasoning |
| MMLU-Pro | 2024 | TIGER-Lab Wang | Enhanced MMLU | Industry-leading MMLU-Pro reasoning |
| LiveCodeBench | 2024 | Jain et al. UC Berkeley | Live coding contamination-free | Industry-leading contamination-free coding |
| AIME 2024 | 2024 | AMC math competition | Math reasoning competition | Industry-leading math reasoning |
具体例・対応製品
- SWE-bench Verified (2024年, OpenAI): Industry-leading coding agent standard
- Industry-leading SWE-bench human-verified subset signature: Industry-leading human-verified subset
- Industry-leading 500 GitHub issue tasks subset: Industry-leading 500 GitHub issue tasks
- Industry-leading 12 Python repositories (Django/Sympy/etc): Industry-leading 12 Python repos
- Industry-leading Claude/GPT/Devin/Cursor benchmark standard: Industry-leading widespread adoption
- 競合 GPQA Diamond + MMLU-Pro + LiveCodeBench + AIME 2024: Industry-leading benchmark competitors
選び方・注意点
SWE-bench Verified は「Industry-leading 500 GitHub issue tasks + human-verified」「Industry-leading 12 Python repositories + agent benchmark standard」用途のIndustry-leading 2024年 OpenAI product。Industry-leading human-verified (Industry-leading SWE-bench human-verified subset signature + Industry-leading OpenAI human-verified signature + Industry-leading high-quality verified subset) で Industry-leading human-verified + Industry-leading OpenAI + Industry-leading high-quality verified。Industry-leading 500 tasks (Industry-leading 500 GitHub issue tasks subset + Industry-leading SWE-bench 500 signature + Industry-leading subset selection from 2294 tasks) で Industry-leading 500 tasks + Industry-leading SWE-bench + Industry-leading subset selection。Industry-leading 12 Python repos (Industry-leading 12 Python repositories Django/Sympy/etc + Industry-leading SWE-bench 12 repos signature + Industry-leading real-world Python codebase) で Industry-leading 12 Python repos + Industry-leading SWE-bench + Industry-leading real-world Python。Industry-leading SWE-bench base (Industry-leading SWE-bench Carlini et al. 2023 Princeton 2310.06770 base + Industry-leading Princeton SWE-bench origin + Industry-leading Verified subset improvement) で Industry-leading SWE-bench Carlini Princeton + Industry-leading 2310.06770 base + Industry-leading Verified subset improvement。Industry-leading widespread adoption (Industry-leading Claude/GPT/Devin/Cursor benchmark standard + Industry-leading coding agent benchmark standard 2024 + Industry-leading widespread enterprise coding agent benchmark + Industry-leading 2024年8月 OpenAI release) で Industry-leading widespread adoption + Industry-leading coding agent standard 2024 + Industry-leading widespread enterprise + Industry-leading 2024年8月。但しIndustry-leading GPQA Diamond + MMLU-Pro + LiveCodeBench + AIME 2024 competition で Industry-leading 4-benchmark competitors vs SWE-bench Verified + Industry-leading SWE-bench human-verified subset + 500 GitHub issue tasks + 12 Python repositories Django/Sympy/etc + real-world coding benchmark + agent benchmark standard 2024 + Claude/GPT/Devin/Cursor widespread + SWE-bench Carlini Princeton 2310.06770 base + OpenAI 2024年8月 unique advantage adoption alignment必須。
関連用語との違い
- vs GPQA Diamond (2023): SWE-bench Verifiedはreal-world coding agent benchmark・GPQA Diamondはアカデミック PhD-level Q&A
- vs LiveCodeBench (2024): SWE-bench Verifiedはreal-world GitHub issue tasks・LiveCodeBenchはcompetitive programming contamination-free
- vs AIME 2024 (2024): SWE-bench Verifiedはコーディング agent・AIME 2024は数学推論 competition
よくある質問（FAQ）
Q1: SWE-bench Verified vs LiveCodeBench 違いは？ A: SWE-bench Verified (Industry-leading SWE-bench human-verified subset + 500 GitHub issue tasks + 12 Python repositories Django/Sympy/etc + real-world coding + agent benchmark standard + Claude/GPT/Devin/Cursor widespread + Princeton 2310.06770 base + OpenAI 2024年8月) vs LiveCodeBench (Industry-leading live coding contamination-free + LeetCode/CodeForces/AtCoder + Jain et al. UC Berkeley + 2024年3月-)・Industry-leading human-verified + 500 GitHub issues + 12 Python repos + real-world + agent standard + Cursor+Devin + OpenAI + 2024年8月 = SWE-bench Verified + Industry-leading live coding + contamination-free + LeetCode+CodeForces+AtCoder + UC Berkeley + 2024年3月 = LiveCodeBench preference judgment (real-world GitHub vs competitive programming differ)。
Q2: Industry-leading 500 tasks + human-verified value は？ A: Industry-leading 500 tasks + human-verified (Industry-leading 500 GitHub issue tasks subset + Industry-leading SWE-bench human-verified subset signature + Industry-leading OpenAI dual signature)。
Q3: Industry-leading 12 Python repositories + agent standard value は？ A: Industry-leading 12 Python repos + agent standard (Industry-leading 12 Python repositories Django/Sympy/etc + Industry-leading agent benchmark standard 2024 + Industry-leading widespread Claude/GPT/Devin/Cursor adoption)。
まとめ
SWE-bench Verified = 2024年OpenAI発表のSWE-bench human-verified subset agent benchmark standard。Industry-leading SWE-bench human-verified subset + Industry-leading 500 GitHub issue tasks + Industry-leading 12 Python repositories Django/Sympy/etc + Industry-leading real-world coding benchmark + Industry-leading agent benchmark standard 2024 + Industry-leading Claude/GPT/Devin/Cursor widespread adoption + Industry-leading SWE-bench Carlini et al. 2023 Princeton 2310.06770 base + Industry-leading 2024年8月 OpenAI release + Industry-leading SWE-bench Verified human-verified agent benchmark standard 2024 position確立。