Building Trust and Experiment Validity

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Topic 8 of 10

Published Jul 27, 2023 · Clip 42:00–1:02:14 (20:14)

Ronnie notes Airbnb's search team A/B tested everything, but company-wide shift to top-down reduced experiments, potentially limiting growth. During COVID, experiments crucial for quick validation amid uncertainty. Emphasizes trust via platforms for safety nets and oracles; early Optimizely's p-value errors eroded trust. Common invalidity: sample ratio mismatch (8% at Microsoft).