
Building and Aligning LLM Judges

Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar
Topic 3 of 4
Published Sep 25, 2025 · Clip 44:40–1:04:55 (20:15)
Guests explain building LLM judges for complex failures like handoffs, emphasizing binary outputs and alignment with human judgments via matrices. They cover code-based vs. LLM-based evals, warn against Likert scales, and share research on criteria drift where rubrics evolve with data. Evals enable unit tests, CI/CD integration, and production monitoring for ongoing quality measurement.