
Evaluating new models for coding

Building Windsurf with Varun Mohan
Topic 1 of 16
Published May 7, 2025 · Clip 1:39–6:23 (4:44)
The host asks about GPT-4.1 support and how Windsurf evaluates new models for coding. Varun explains that models are non-deterministic and that the team, drawing on autonomous vehicles experience, built extensive evaluation suites. These test end-to-end task completion plus retrieval accuracy, edit accuracy, and redundant changes. They use open-source repository commits with ground truth to create multi-layered programmatic tests, allowing evaluation across tens of thousands of repositories within minutes instead of human AB testing.