Brian Tolkin on Product-Ops Integration, Experimentation at Scale, and the Kernel of Truth

Brian Tolkin on Product-Ops Integration, Experimentation at Scale, and the Kernel of Truth

transcriptlennys-podcastproduct-opsuberopendoorexperimentationoperationsmarketplace

Brian Tolkin on Product-Ops Integration, Experimentation at Scale, and the Kernel of Truth

Source: Lenny’s Podcast Speaker: Brian Tolkin Date: ~2023–24 Link: https://www.lennysnewsletter.com/p/scaling-uber-and-opendoor-brian-tolkin

Key ideas

  • Twin turbine jet plane — product + ops. Product and operations are not competitors but complementary engines; running on one is possible but inefficient. Ops teams have irreplaceable advantages: faster iteration at small scale, rich qualitative insight, local knowledge a SF-based PM can never replicate. Product teams have tech leverage and scaling ability. Respect the asymmetry: early days prioritise where tech leverage is greatest (Uber’s dispatching and pricing systems), let ops carry the rest; as scale grows, systematically automate the ops tasks that have hit their ceiling. Driver onboarding trajectory: 1-on-1 → small classroom → video → OCR auto-validation — each switch triggered when ops iteration was no longer viable.
  • Product operations as a function. At Uber, product ops was a formal function bridging a centralised SF product team and a globally distributed ops team — bidirectional feedback loop for how features get deployed in markets and how market insight shapes features. Product ops people reported into operations but physically sat with and operated like members of the product team. Distinct from traditional product management; solves the localisation and feedback problem at scale.
  • Product reviews: not a firing squad. Two explicit goals: (1) accountability and informing; (2) making the product better. Tone set by leader: probe and ask questions rather than mandate; acknowledge that the team closest to the problem has the most context. Template: context → problem → potential solution → risks/pre-mortem → measurement of success. Stage-gated: ideation artefacts look very different from near-ship artefacts. Keep attendance under 10; wide document distribution; artefacts double as onboarding material for new hires.
  • Experimentation under low sample sizes. Always run a power analysis before starting an A/B test — be honest whether the runtime and MDE are acceptable. Alternatives when canonical A/B is not viable: diff-in-diff with observational data, sister/twin city comparison, geo segmentation, reducing confidence threshold to 80%, long-term holdout experiments. If none of these is available, trust intuition — but build a feedback loop (support ticket volume, feature adoption) to validate afterwards. Never pretend insignificant results are significant.
  • Finding the kernel of truth. Product is finding the kernel of truth in a sea of ambiguity and signals. Ideas come from everywhere (CX, customers, executives, field visits, competitors), but the job is to identify what really moves the customer forward and say no to the rest. At Uber early: rider ↔ driver dispatch + pricing were the only levers that mattered; all other investment was noise. Same discipline at Opendoor: identify where tech uniquely creates leverage and be explicit that everything else will burn for now. Write every input down — it validates that contributors were heard and creates a searchable backlog.

Overview

Brian Tolkin joined Uber as employee ~100 in 2012 (ops team), moved into product, led uberPOOL from launch to global rollout, and pioneered the product operations function. He is now head of product and design at Opendoor. The episode covers: product + ops partnership philosophy; the product operations function Tolkin helped formalise at Uber; how to run effective product reviews without creating a firing squad culture; experimentation tactics when transaction volumes are low; and how to identify the kernel of truth amid signal noise. Rich in practitioner war stories: surge pricing’s human-in-the-loop origins, the uberPOOL China launch (no sleep, Chengdu pancakes), Opendoor’s COVID pivot to virtual tours, and Zillow’s failed entry into iBuying.