
Benchmarks Flaws and Human Evaluations

The $1B Al company training ChatGPT, Claude & Gemini on the path to responsible AGI | Edwin Chen
Topic 4 of 8
Published Dec 7, 2025 · Clip 17:38–22:28 (4:50)
Benchmarks are untrustworthy due to flaws like wrong answers and ease of gaming via objective, hill-climbable metrics that don't correlate with real-world messiness. Models excel on IMO but fail at PDF parsing because benchmarks lack ambiguity. True progress is measured via deep human evaluations by domain experts across tasks like physics research or coding, assessing accuracy and depth beyond casual vibes.