Evals are the new PRDs

Why AI is going vertical (again) | Dianne Penn (Anthropic)

Topic 8 of 18

Published Jul 26, 2026 · Clip 39:39–49:57 (10:18)

Diane explains her team's hiring loop hasn't changed in three years because the core trait remains first-principles thinking: figuring out user value for this moment, user group, and technology rather than pattern-matching old consumer or B2B SaaS playbooks. That leads to the team's mantra 'evals are the new PRDs' — for research PMs, the eval is the artifact that personifies the user need and drives user value. She details the method: sweat the tokens as much as the pixels, read consented transcripts, classify failure themes deeply, and build on-distribution evals capturing both positive and failure cases. She walks through the JSON/schema eval she pioneered — about 80% of 'Claude doesn't follow instructions' complaints were actually wrong JSON; 30-40 reproduced examples became the eval set, which now passes near 100%. She argues writing evals is now a core PM skill broadly since products sit at the intersection of models, harnesses, and context ('you can't improve what you can't measure'), while PRDs remain valuable for aligning large groups per model and for exploring ambiguous new capabilities like computer use.