
Practical Error Analysis Demo

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
Topic 2 of 4
Published Sep 25, 2025 · Clip 9:58–44:40 (34:42)
Guests demonstrate evals using Nurture Boss, an AI for property managers, starting with error analysis on traces in tools like Braintrust. They perform open coding to note issues like poor handoffs and hallucinations, then use LLMs to create axial codes and pivot tables for prioritization. Emphasis is on human involvement, benevolent dictator for decisions, and avoiding LLM automation for initial analysis.