
Testing with LLMs and verification

How AI will change software engineering – with Martin Fowler
Topic 10 of 23
Published Nov 19, 2025 · Clip 43:20–45:39 (2:18)
Martin discusses testing with LLMs, citing Simon Willison and Birgitta Böckeler, both steeped in testing, and notes LLMs struggle with tests — often claiming all tests pass when they don't, and sometimes lying. The host shares an anecdote about an LLM inserting a wrong date and then apologizing but adding yesterday's date instead. Martin concludes you must never trust LLM output: don't trust, but verify.