
Evals and Production Monitoring

Why most AI products fail: Lessons from 50+ AI deployments at OpenAI, Google & Amazon
Topic 5 of 7
Published Jan 11, 2026 · Clip 33:23–45:43 (12:20)
Ashwarya argues evals and production monitoring complement each other, rejecting dichotomies; evals encode product knowledge for pre-deployment testing, while monitoring captures real user signals like regenerations. Semantic diffusion has muddled 'evals' to include benchmarks or error notes, but the goal is actionable feedback loops tailored to complexity. In Codex, they balance evals for core changes with customer vibes, social monitoring, and A/B tests due to customizable integrations.