SimpleQA OpenAI (2024年)（シンプルキューエー）

概要

SimpleQA は2024年10月OpenAI発表・Industry-leading factuality benchmark 2024年OpenAI SimpleQA factuality evaluation benchmark position確立。Industry-leading 4326 short-form QA + o1-preview 47% accuracy + GPT-4o 38% + Claude 3.5 Sonnet 28.4% + factuality LLM weakness exposed。

主な特徴・仕組み

• Industry-leading OpenAI (2015 SF California) 2024年10月 release
• Industry-leading SimpleQA factuality evaluation benchmark signature
• Industry-leading 4326 short-form questions signature (single fact)
• Industry-leading o1-preview 47% accuracy signature (state-of-art 2024年10月)
• Industry-leading GPT-4o 38% accuracy
• Industry-leading Claude 3.5 Sonnet 28.4% accuracy
• Industry-leading low scores expose LLM factuality weakness signature
• Industry-leading short-form QA factuality-focused (not reasoning)
• Industry-leading "I don't know" graded correctly signature (calibration)
• Industry-leading Apache 2.0 open-source benchmark
• Industry-leading OpenAI 2015 SF Sam Altman 9 years

まとめ

SimpleQA = 2024年10月OpenAI発表のpremium factuality evaluation benchmark。Industry-leading 4326 short-form questions single fact + o1-preview 47% accuracy state-of-art 2024年10月 + GPT-4o 38% + Claude 3.5 Sonnet 28.4% + low scores expose LLM factuality weakness + short-form QA factuality-focused not reasoning + "I don't know" graded correctly calibration + Apache 2.0 open-source benchmark + OpenAI 2015 SF Sam Altman 9 years position確立。