
Handling Imperfect AI Models

A conversation with OpenAI's CPO Kevin Weil, Anthropic's CPO Mike Krieger, and Sarah Guo
Topic 4 of 9
Published Nov 6, 2024 · Clip 12:00–16:01 (4:01)
Guests argue that even 60% model accuracy can be valuable if designed for human-in-the-loop, citing examples like GitHub Copilot which succeeded despite imperfections by saving time on coding. They note lumpy performance across tasks, where pilots yield varied feedback from customers. Real-world deployment reveals surprises beyond internal evals, emphasizing the need to adapt products for graceful failures. Progression involves product burdens like UI for corrections rather than waiting for perfect models.