Evaluating AI Systems

AI Engineering with Chip Huyen

Topic 9 of 14

Published Feb 5, 2025 · Clip 36:58–48:00 (11:02)

Chip calls evaluation the trillion-dollar question. As AI gets smarter, humans struggle to evaluate it (terminator analogy: Terence Tao comparing using o1 to advising a mediocre but not completely stupid student). She covers approaches: functional correctness (which is why coding is a popular use case), AI-as-a-judge, and comparative evaluation. She stresses that evaluation requires discipline — talking to users and measuring what matters, with examples of meeting summarization and a tax-firm chatbot where users didn't like typing. She recommends manual data inspection (citing Greg Brockman) and consistent human evaluations to correlate with automated metrics.