Importance of AI Evaluations

A conversation with OpenAI's CPO Kevin Weil, Anthropic's CPO Mike Krieger, and Sarah Guo

Topic 5 of 9

Published Nov 6, 2024 · Clip 16:01–20:00 (3:59)

Evals unlock model potential beyond base intelligence, teaching specifics via prompts and fine-tuning to boost accuracy from 60% to 85%. Guests share how Anthropic uses Claude to automate eval writing and grading, stressing clear success definitions. PM roles are merging research and product, requiring eval skills; bootcamps train teams on good vs. bad evals. Intuition builds by examining data, refining graders, and using models to generate samples.