Ramesh Johari on Marketplaces and Data Science

Ramesh Johari on Marketplaces and Data Science

marketplacesdata-scienceexperimentationrating-systems

Ramesh Johari on Marketplaces and Data Science

Speaker: Ramesh Johari Source: Lenny’s Podcast Date: c. 2023

Ramesh Johari, Stanford professor and longtime advisor to Airbnb, Uber, Stripe, Bumble, and Upwork, provides a rigorous framework for thinking about marketplace businesses — what they sell, how data science powers them, and where founders go wrong.

Key ideas

  • Marketplaces sell the removal of friction, not the underlying goods. Both sides (buyers and sellers) are customers of the platform. This framing shapes strategy: the business model should reflect the friction being removed at each stage of growth.
  • A marketplace never starts as a marketplace. Before achieving scaled liquidity on both sides, the real question is: what is my value proposition when I have no scale? UrbanSitter started by solving the credit-card payment friction; oDesk started by solving remote work verification trust.
  • Prediction ≠ decision-making. Machine learning finds correlations; decisions require causal inference. Sending promotions to high-LTV customers is only correct if the promotion causes more spend — not merely because those customers already have high LTV.
  • The whac-a-mole principle. Most marketplace changes that matter create winners and losers by reallocating inventory and attention. Metrics will move around unpredictably — the question is always whether the winners matter more than the losers.
  • Rating inflation and distributional fairness. Averaging ratings disadvantages new entrants. A first negative rating can drop expected revenue by ~8% and correlates with platform exit. Bayesian priors can mitigate this. Double-blind reviews (Airbnb’s model) increase review rates and information quality.

Detailed notes

What marketplaces sell

Transaction costs prevent markets from clearing — riders can’t find drivers, guests can’t find hosts. Platforms remove those costs. “That’s what you’re paying them for.” Implication: the platform’s monetisation model should reflect the stage of friction being removed. oDesk’s early commission model became misaligned as long-term relationships matured and disintermediation grew attractive.

Data science flywheel

Three stages: (1) finding potential matches (search, recommendation, ranking); (2) making matches (triaging applicants, ranking algorithms); (3) learning from matches (ratings, feedback, passive behavioural data). Each stage informs the next. Good data scientists think in terms of causal decisions — which ranking algorithm produces more bookings? — not merely predictive accuracy.

Experimentation

Bayesian A/B testing incorporates past experiments into priors, rewarding learning not just wins. The “winners and losers” framing incentivises incremental experiments and long run times, crowding out bold bets. Suggests allocating some percentage of resources to tail experiments even expecting failure.

Rating systems

No platform has truly “nailed” ratings. Rating inflation results from reciprocity and norming. Renorming techniques (e.g. “exceeded expectations” labels; comparative framing) help. The “sound of silence” — not leaving a rating — carries information; effective-percent-positive metrics that include non-responses are more predictive of seller performance.

AI and data science

AI expands the frontier of hypotheses and explanations, increasing the importance of human judgment to filter and prioritise — not reduce it. The number of things worth testing grows astronomically; the human question of “what matters?” becomes harder, not easier.