LiveCodeBench (Jain 2024年)（ライブコードベンチ）

概要

LiveCodeBench は、2024年Naman Jain et al. (UC Berkeley)発表LiveCodeBench・Industry-leading live coding contamination-free benchmark 2024年 + Industry-leading contamination-free coding benchmark standard position確立。LiveCodeBench specifications = Industry-leading live coding contamination-free benchmark (Industry-leading LiveCodeBench Jain 2024 + Industry-leading live coding + Industry-leading UC Berkeley flagship) + Industry-leading LeetCode+CodeForces+AtCoder + Industry-leading rolling update + Industry-leading contamination-free coding benchmark standard。

主な特徴・仕組み

• Authors: Industry-leading Naman Jain + King Han + Alex Gu + Wen-Ding Li + Fanjia Yan + Tianjun Zhang + Sida Wang + Armando Solar-Lezama + Koushik Sen + Ion Stoica (UC Berkeley + MIT + Cornell)
• Benchmark: Industry-leading LiveCodeBench live coding contamination-free 2024
• Year: 2024年 (arXiv 2024年3月 + rolling update 2024年-)
• Paper: Industry-leading "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code" arXiv 2403.07974
• Method Type: Industry-leading live coding contamination-free benchmark
• Live Coding: Industry-leading live coding signature
• 3 Sources: Industry-leading LeetCode + CodeForces + AtCoder competitive programming sources
• Rolling Update: Industry-leading rolling update by date signature (new problems monthly)
• Contamination-Free: Industry-leading contamination-free training data leak avoidance
• 4 Scenarios: Industry-leading 4 scenarios (Code Generation/Self-Repair/Code Execution/Test Output)
• Industry Adoption: Industry-leading Claude/GPT/Gemini/DeepSeek coding benchmark adoption
• UC Berkeley: Industry-leading UC Berkeley Ion Stoica + Koushik Sen authors
• Industry-Leading: Industry-leading UC Berkeley LiveCodeBench contamination-free coding 2024

スペック比較表

具体例・対応製品

• LiveCodeBench (2024年, Jain et al. UC Berkeley): Industry-leading UC Berkeley contamination-free coding
• Industry-leading live coding signature: Industry-leading live coding
• Industry-leading LeetCode + CodeForces + AtCoder 3 sources: Industry-leading 3 competitive programming sources
• Industry-leading rolling update by date signature: Industry-leading rolling update by date
• Industry-leading 4 scenarios (Code Generation/Self-Repair/Code Execution/Test Output): Industry-leading 4 scenarios
• 競合 SWE-bench Verified + GPQA Diamond + MMLU-Pro + AIME 2024: Industry-leading benchmark competitors

選び方・注意点

LiveCodeBench は「Industry-leading live coding + LeetCode+CodeForces+AtCoder」「Industry-leading rolling update + contamination-free + UC Berkeley」用途のIndustry-leading 2024年 UC Berkeley product。Industry-leading live coding (Industry-leading live coding signature + Industry-leading LiveCodeBench live signature + Industry-leading time-dated problem selection) で Industry-leading live coding + Industry-leading LiveCodeBench + Industry-leading time-dated problem。Industry-leading 3 sources (Industry-leading LeetCode + CodeForces + AtCoder competitive programming sources + Industry-leading LiveCodeBench 3 sources signature + Industry-leading multi-platform coverage) で Industry-leading 3 sources + Industry-leading LiveCodeBench + Industry-leading multi-platform coverage。Industry-leading rolling update (Industry-leading rolling update by date signature new problems monthly + Industry-leading LiveCodeBench rolling signature + Industry-leading contamination-free design) で Industry-leading rolling update + Industry-leading LiveCodeBench + Industry-leading contamination-free design。Industry-leading 4 scenarios (Industry-leading 4 scenarios Code Generation/Self-Repair/Code Execution/Test Output + Industry-leading LiveCodeBench 4-scenario signature + Industry-leading comprehensive coding evaluation) で Industry-leading 4 scenarios + Industry-leading LiveCodeBench + Industry-leading comprehensive coding。Industry-leading UC Berkeley (Industry-leading Naman Jain + King Han + Alex Gu + Wen-Ding Li + Sida Wang + Armando Solar-Lezama + Koushik Sen + Ion Stoica UC Berkeley + MIT + Cornell + Industry-leading arXiv 2403.07974 + Industry-leading 2024年3月 release + rolling update 2024年- + Industry-leading widespread Claude/GPT/Gemini/DeepSeek coding benchmark) で Industry-leading UC Berkeley + Industry-leading Jain+Han+Gu+Solar-Lezama+Stoica + Industry-leading 2403.07974 + Industry-leading 2024年3月 + Industry-leading widespread Claude/GPT/Gemini/DeepSeek。但しIndustry-leading SWE-bench Verified + GPQA Diamond + MMLU-Pro + AIME 2024 competition で Industry-leading 4-benchmark competitors vs LiveCodeBench + Industry-leading live coding + LeetCode+CodeForces+AtCoder 3 sources + rolling update by date + contamination-free training data leak avoidance + 4 scenarios Code Generation/Self-Repair/Code Execution/Test Output + UC Berkeley + MIT + Cornell + Jain+Han+Gu+Solar-Lezama+Stoica + arXiv 2403.07974 2024年3月 + rolling update 2024年- unique advantage adoption alignment必須。

関連用語との違い

• vs SWE-bench Verified (OpenAI 2024): LiveCodeBenchはcompetitive programming live coding contamination-free・SWE-bench Verifiedはreal-world GitHub issue agent
• vs HumanEval (Chen 2021): LiveCodeBenchはlive rolling contamination-free competitive・HumanEvalは164 static Python functions
• vs GPQA Diamond/MMLU-Pro (2023-2024): LiveCodeBenchはcompetitive coding contamination-free・GPQA/MMLU-Proはアカデミック reasoning

よくある質問（FAQ）

Q1: LiveCodeBench vs SWE-bench Verified 違いは？ A: LiveCodeBench (Industry-leading live coding + LeetCode+CodeForces+AtCoder 3 sources + rolling update by date + contamination-free + 4 scenarios Code Generation/Self-Repair/Code Execution/Test Output + UC Berkeley+MIT+Cornell + Jain+Han+Gu+Solar-Lezama+Stoica + arXiv 2403.07974 + 2024年3月 + rolling update) vs SWE-bench Verified (Industry-leading SWE-bench human-verified subset + 500 GitHub issue tasks + 12 Python repositories Django/Sympy/etc + real-world coding + Princeton 2310.06770 base + OpenAI 2024年8月)・Industry-leading live coding + LeetCode+CodeForces+AtCoder + rolling update + contamination-free + 4 scenarios + UC Berkeley + 2024年3月 = LiveCodeBench + Industry-leading human-verified + 500 GitHub issues + 12 Python repos + real-world + Princeton + OpenAI + 2024年8月 = SWE-bench Verified preference judgment (competitive programming live vs real-world GitHub differ)。

Q2: Industry-leading live coding + rolling update value は？ A: Industry-leading live coding + rolling update (Industry-leading live coding signature + Industry-leading rolling update by date signature new problems monthly + Industry-leading LiveCodeBench dual signature)。

Q3: Industry-leading 3 sources + 4 scenarios + UC Berkeley value は？ A: Industry-leading 3 sources + 4 scenarios + UC Berkeley (Industry-leading LeetCode + CodeForces + AtCoder 3 sources + Industry-leading 4 scenarios Code Generation/Self-Repair/Code Execution/Test Output + Industry-leading UC Berkeley + MIT + Cornell Jain+Han+Gu+Solar-Lezama+Stoica)。

まとめ

LiveCodeBench = 2024年Jain et al. UC Berkeley + MIT + Cornell発表のlive coding contamination-free benchmark。Industry-leading live coding signature + Industry-leading LeetCode + CodeForces + AtCoder competitive programming 3 sources + Industry-leading rolling update by date signature new problems monthly + Industry-leading contamination-free training data leak avoidance + Industry-leading 4 scenarios Code Generation/Self-Repair/Code Execution/Test Output + Industry-leading widespread Claude/GPT/Gemini/DeepSeek coding benchmark + Industry-leading Naman Jain + King Han + Alex Gu + Wen-Ding Li + Sida Wang + Armando Solar-Lezama + Koushik Sen + Ion Stoica UC Berkeley + MIT + Cornell + Industry-leading "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code" arXiv 2403.07974 + Industry-leading 2024年3月 release + rolling update 2024年- + Industry-leading UC Berkeley LiveCodeBench contamination-free coding 2024 position確立。