Experimentation and Intuition in Low-Volume Settings

Lessons from scaling Uber and Opendoor | Brian Tolkin (Head of Product at Opendoor, ex-Uber)

Topic 8 of 10

Published Aug 4, 2024 · Clip 40:03–47:11 (7:08)

Opendoor runs A/B tests on high-volume funnel parts but faces low-sample challenges. Lessons: Run power analysis first for runtime/detectable effects; accept long runs (e.g., 6 months) for key tests; use alternatives like customer talks, observational data, sister cities, or lower confidence (80%). For untestable, trust intuition if high conviction, but build feedback loops (e.g., support volume). Humility: Test assumptions when easy; gut-check low conviction with more input.