Testing and Engineering Practices for ML

Building Effective and Thriving Machine Learning Teams - David Tan & Dave Colls

Topic 7 of 9

Published Jan 27, 2025 · Clip 39:23–48:25 (9:02)

The guests discuss missing engineering disciplines in ML. David Tan emphasizes test automation, recounting an LLM project that grew from near-zero coverage to a test pyramid plus model eval suites, enabling a Snyk dependency-upgrade PR to be merged confidently in 15 minutes. He links lack of design to lack of tests: without tests, refactoring is scary. David Colls extends the test pyramid with a data dimension—canaries, sample tests, and full-data tests—for faster feedback, and describes visualization-driven development for exploratory analysis. David Tan shares how they shifted LLM testing left by asserting on intent rather than vocabulary using PyHamcrest-style matchers.