P-Values, Getting Started, and Speed

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Topic 9 of 10

Published Jul 27, 2023 · Clip 1:02:14–1:14:16 (12:02)

P-value ≠ treatment superiority probability; common error—use Bayes for false positive risk (26% at Airbnb despite p<0.05). Replicate borderline results. To start: Consult internals, build/buy platform (third-party viable now). Culture: Beachhead in frequent-launch teams with clear OEC. Platforms reduce costs to zero via self-service, automation (e.g., CUPED for 20-50% speed).