
Evals Debates, Misconceptions, and Tips

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
Topic 4 of 4
Published Sep 25, 2025 · Clip 1:04:55–1:46:32 (41:37)
Guests address debates: evals aren't just tests but include error analysis and monitoring; coding agents rely on developer expertise but still use evals indirectly. AB tests complement evals but shouldn't replace data-grounded prioritization. Misconceptions include over-relying on AI tools or skipping data; tips emphasize fun in analysis, using AI for synthesis, and custom tools. They promote their Maven course for deeper skills.