
Taming AI randomness and quality evaluation

Building AI-Powered Podcast Learning 🎧 — with Kevin Smith
Topic 4 of 7
Published Jun 27, 2025 · Clip 21:23–31:37 (10:13)
Kevin discusses handling the randomness of AI models, using the analogy of managing a thousand interns: retrying tasks, having multiple models produce candidates, and using a judge LLM to pick the best output. He recounts the history from GPT-3 era JSON parsing struggles (solved by sending errors back to the model, and later by JSON mode/structured outputs) to the current challenge of output quality, illustrated by the Snipd Wrapped quote extraction where 50% of quotes were boring until they used multiple candidate models plus a judge. On evaluating model swaps, he admits they rely on 'vibe evaluations' and dogfooding, reverting changes when quality drops.