Ronny Kohavi
Widely regarded as the world expert on A/B testing and online controlled experiments. Spent his career building experimentation culture and platforms at Amazon (Director of Data Mining and Personalisation), Microsoft (Corporate VP, led Bing’s experimentation platform and Microsoft Experimentation Platform), and Airbnb (VP and Technical Fellow, search relevance). Author of Trustworthy Online Controlled Experiments (Cambridge University Press) — all proceeds donated to charity. Full-time advisor and instructor; teaches a cohort course on Maven.
Appearances
Key ideas
Test everything. Any code change, any feature, even small bug fixes can produce surprising, unexpected results. The marginal cost of running an experiment should approach zero once a platform matures. At Microsoft/Bing, ~20–25,000 experiments per year at peak.
Most ideas fail. 66% failure at Microsoft overall; 85% at Bing; 92% in Airbnb search. This is not a reflection of poor teams — Google Ads, Booking, and others report similar rates. The universal beginner’s error is expecting to be different.
OEC and causal thinking. The overall evaluation criterion must be causally predictive of lifetime value. Any metric that can be gamed (raw revenue, time on site) without a countervailing guardrail is the wrong OEC.
Trust is the foundation. Optimizely’s early real-time P-value monitoring inflated false-positive rates to ~30%, demonstrating how quickly experimentation trust can collapse. Ramesh Johari at Stanford later consulted to fix the methodology.
Structured narrative. Learned at Amazon: write a six-pager narrative instead of a slide deck. Prose forces rigour; annotations persist as institutional memory.