Judd Antin on the User Research Reckoning, Middle-Range Research, and User-Centred Performance

Source: Lenny’s Podcast Speaker: Judd Antin Link: Episode

Judd Antin built the user research practice at Meta and was head of research — and later head of the UX design studio — at Airbnb. His direct reports have gone on to lead research at Figma, Notion, Slack, Robinhood, and Duolingo. In this episode Antin makes the case that the wave of researcher layoffs following the ZIRP era is a symptom of a broken integration model, not evidence that research is unimportant. His diagnosis is unusually sharp, and his prescriptions are specific.

Key ideas

The reckoning is structural, not disciplinary. The mass layoffs of UX researchers in 2022–23 were not caused by researchers doing bad work. They were caused by companies hiring researchers into service functions with poorly defined mandates, then discovering that middle-range research — the bulk of the workload — was not moving business metrics.
Macro, middle-range, and micro research. Antin’s three-tier framework: macro research is strategic, competitive, and forward-looking (TAM studies, concept cars, annual planning inputs); micro research is tightly scoped usability and conversion work with direct business impact; middle-range research asks interesting questions about how users think and feel but is neither strategic nor immediately actionable. Most companies do too much middle, too little micro, and far too little macro.
User-centred performance. Antin names the phenomenon of performing user-centricity — running research not to change a decision but to signal customer obsession. The exec listening session, the validation study at the end of a process, the focus group the PM wants to attend — these are performances. The tell is that no one expects to be wrong. One of Antin’s mantras: ‘We don’t validate, we falsify.’
The multimillion-dollar button. A concrete Airbnb example: research revealed that users were abandoning the purchase funnel because the call-to-action button copy felt like it initiated a purchase when it only advanced to the next step. Seven characters changed. Conversion rate improved by approximately 1%, worth millions in revenue. Done in 48 hours.
NPS is the best example of the marketing industry marketing itself. The Likelihood to Recommend question uses an 11-point scale, is largely unlabelled, correlates poorly with business outcomes, and is idiosyncratic enough that cross-company benchmarking is invalid. Customer satisfaction (CSAT) — ‘How satisfied are you with your experience?’ — has better psychometric properties and stronger correlation with retention.

Topics covered

The ZIRP-era hiring boom and why researchers were set up to fail
Macro, middle-range, and micro research: definitions and examples
Why middle-range research triggers post-hoc bias and ‘we already knew that’
User-centred performance: what it looks like and why it’s epidemic
The five tools of a strong researcher: formative, evaluative, survey design, applied statistics, SQL/prompt engineering
Why researchers should read the quarterly earnings call
The researcher-to-PM relationship: the most troubled partnership in product
Five PM tropes about research that drive researchers nuts (and why they’re mostly wrong)
Research speed: good research does not slow teams down — middle-range research does
The multimillion-dollar button: what fast micro research looks like
The ‘super hider’ Facebook story: N=1 blows up an engineering team’s model
Wisdom of the crowd: when diverse intuitions beat individual gut
NPS vs. CSAT: why NPS is methodologically indefensible
What researchers and companies should do next: five tools, business fluency, integrated process

Signals

User-Centred Performance — novel concept; see concept page