Experiment Failure Rates and Incremental Gains

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Topic 3 of 10

Published Jul 27, 2023 · Clip 10:54–15:18 (4:24)

Ronnie explains that big wins are rare, with most progress from incremental gains adding up, like Bing's 2% annual relevance improvement or Airbnb's 6% revenue lift from 250 experiments. He shares failure rates: 66% at Microsoft, 85% at Bing, 92% at Airbnb, noting 10% aborts due to bugs and the need for iteration.