Reading Notes

Annie Duke on Better Decisions, Kill Criteria, and When to Quit

Source: Annie Duke on Better Decisions, Kill Criteria, and When to Quit

Notes — Annie Duke on Better Decisions, Kill Criteria, and When to Quit

Four questions [Adler frame]

Q1 — What is it about? How to close the gap between knowing about cognitive biases and actually deciding better. Duke — former professional poker player, author of Thinking in Bets and Quit, special partner at First Round Capital — argues the single highest-leverage move is to make implicit intuition explicit, so it can be examined, scored against outcomes, and improved. The episode applies this to meetings (the nominal group technique), to long-horizon bets (feedback-loop design), to launches (pre-mortems with kill criteria), and to quitting (why we always quit too late). Plus a tribute to Daniel Kahneman and a parenting digression that doubles as decision-tool demonstration.

Q2 — How is it argued? From practice rather than from the literature she could easily cite. Her central evidence is the five-year First Round Capital experiment: structured investment rubrics scored one-to-seven, explicit probability forecasts (chiefly ‘will this fund at Series A?’), and a now-large dataset that lets the firm tell each partner where their confident table-pounding predicts outcomes and where it predicts nothing. She backs the meeting argument with Kahneman’s hiring work (unstructured-to-structured lifted hit rate from ~50% to ~65%) and the quitting argument with the Stewart Butterfield Glitch→Slack story.

Q3 — Is it true? The core mechanism — that making intuition explicit lets you discover when it is wrong — is sound and self-evidently right; the First Round data, where equally vehement table-pounding is sometimes predictive and sometimes correlated with nothing, is its strongest support. The ‘no such thing as a long feedback loop’ claim is well-argued: a billion-dollar exit has never happened without a Series A, so Series A funding (~16 months) is a necessary leading indicator. The ‘you’ve already quit too late’ claim rests on cited bias research (sunk cost, endowment, identity) plus the Butterfield anecdote — persuasive but one vivid case. [?] The 50%→65% hiring figure is attributed to Kahneman’s work but not pinned to a specific study here.

Q4 — What of it? The nominal group technique is the most portable takeaway: gather opinions independently and asynchronously before the meeting, reserve the meeting for discussion (especially of disagreement), and decide outside the room via a single DRI or private vote. Kill criteria turn a pre-mortem from catharsis into pre-commitment. Feedback-loop design lets any long-horizon decision be scored in months. All three are deployable next week with a survey tool and a spreadsheet.

Glossary

Make the implicit explicit — Duke’s master move. Convert intuition (‘I know a great PM when I see one’) into stated criteria, rubrics, and explicit forecasts. Only an explicit model can be examined, discussed, compared to outcomes, and corrected. Not anti-intuition — intuition is sometimes right, but you cannot find out when it is wrong unless it is explicit.

Nominal group — a group that, at the moment of generating opinions, works independently and asynchronously rather than talking at once. The fix for crosstalk, the loudest voice, and the most-confident-voice dominating.

Discover / discuss / decide — the three things people wrongly cram into a meeting. Only discuss belongs in the room. Discover (eliciting opinions, forecasts, estimates) happens independently beforehand; decide happens after, via a DRI or private vote.

Resulting — judging a decision’s quality by its outcome rather than its process. Duke’s running example: feeling like a great parent because the first child avoided trouble, then learning from the others it was temperament, not parenting.

Mediating judgments — sub-judgments made before an overall rating, which together form an implied definition. Rating ‘competitive landscape’ before rating ‘market strength’ forces a shared definition of what ‘market’ means across raters.

Feedback loop (chosen length) — Duke’s claim that no feedback loop is inherently long. You shorten it by identifying leading indicators necessary but not sufficient for the eventual outcome, and scoring your forecast of those. VC example: fund-at-Series-A in ~16 months, not exit in ten years.

Pre-mortem — imagining, before launch, that the project has already failed, and listing the early signals that foretold it. Alone it rarely changes the plan.

Kill criteria — the operational payload of a pre-mortem: each predicted failure-signal is paired with a pre-committed action (kill, or a specific test then kill) decided before sunk-cost and status-quo bias take hold.

Sunk cost — weighing what you have already spent when deciding whether to continue. Waste is prospective: if you would not start this today, everything you put in going forward is the real waste.

Endowment effect — valuing what you own above identical things you do not. Acute in product: we build things, then over-value them and resist killing them.

Mental time travel — projecting forward (‘how will I feel about this in ten years?’) to shrink the focusing-effect of the present moment. Used on her children; general decision tool.

Nevertheless — the leadership word that lets someone feel fully heard and still not get their way: ‘I hear you, and your input was incorporated. Nevertheless, this is the path.’

Adversarial collaboration — Kahneman’s method (which he helped invent): find someone who disagrees with you and co-design the study that would resolve the dispute. Example: Kahneman–Deaton’s $75k money-happiness ceiling, later reconciled with Killingsworth’s contrary finding by joint work with Barb Mellers.

Key frameworks

Make the implicit explicit [§ How much can decisions improve?]

Three populations: those who do not know the methods, those who know but do not do them, and the tiny last group who know and do. The payoff sits with the last group, so the edge is large precisely because almost nobody does the work. Kahneman’s hiring result anchors the magnitude — structuring the interview and using intuition only after the rubric lifted hit rate from ~50% to ~65%.

The nominal group technique [§ The one change that sticks]

The easiest high-impact change Duke can get companies to adopt:

  1. Discover independently, beforehand. Email each participant the question (brainstorm, forced-rank, or forecast with a three-to-five-sentence rationale). ‘Don’t reply all — send it back to me.’ For repeated decisions, collect via a form that hides others’ answers.
  2. Circulate, then discuss. Share the collected opinions; reserve the meeting for discussion, focusing on disagreement rather than double-clicking on the ~80% that already agrees.
  3. Decide outside the room. A single decision-maker (her preference) or a private vote.
  4. Facilitate by reflecting, not agreeing. The facilitator never says ‘I agree’ — only reflects back (‘what I heard you say is…’) until the speaker confirms. People feel heard and become endowed to the decision.
  5. Drop ‘alignment’. Ten different people will not leave a room agreeing; expecting it is both unreal and coercive. Aim to convey views, not convince. Independence preserves the true spread of opinion — a junior ‘four weeks’ estimate survives next to a senior ‘three months’, and both learn.

Feedback-loop design [§ Decision quality on a ten-year horizon]

‘There is no such thing as a long feedback loop.’ For any long-horizon outcome, identify necessary-but-not-sufficient leading indicators and forecast them at the decision point. In VC, a >$1bn exit has never occurred without Series A funding, so Series A (~16 months) is a measurable, traceable signal. A long loop is often psychological cover: if you can never be proven wrong, a lucky early Uber cheque can pass for skill indefinitely. Humans trade the long run for feeling right now — the tighter the loop, the more often you risk feeling wrong, which is why people avoid tightening it.

Pre-mortem plus kill criteria [§ Pre-mortems]

A pre-mortem alone (work she did with Maurice Schweitzer and Linnea Gandhi) rarely changes the plan. The value is setting kill criteria: convert each predicted failure-signal into a pre-committed action. Worked example from a sales team prompted to imagine a six-month RFP that died:

  • RFP clearly written with a competitor in mind → ask directly whether they are working with a competitor and how far along; kill or pursue on the answer.
  • Prospect won’t demo, only wants to talk price → kill (you are a box-checking exercise).
  • No decision-maker after the first meetings → offer executive alignment; kill if declined.

Pre-committing the action removes the in-the-moment hope that you will act rationally when the signal appears — ‘why there are many people who climb Everest in the middle of a blizzard’.

When to quit [§ Quitting]

By the time you are thinking about quitting, you are probably already late. The same uncertainty that makes starting hard (luck plus hidden information — hence Bezos’s 70% rule) makes stopping hard. Three biases compound: sunk cost, endowment, and identity (internal and external). Richard Thaler: most people won’t quit until it ‘isn’t a decision’ — the money is gone, the crevasse already reached. Quitting late is socially safe (‘I had no choice’); quitting early invites ‘what?’. The Butterfield story: Glitch had $6M in the bank, great PR, and 5,000 diehards — but 95–99 of every 100 new users left in five minutes. After a six-week paid-marketing test (growing 6–7% week over week), back-of-envelope maths showed CAC would only rise as the gaming market saturated, so it was not a venture-scale business. He shut it, rolled investors into the internal chat tool the team had built, and that became Slack. The invisible cost of persisting is the opportunity you cannot see until you quit: ‘he couldn’t see Slack until he quit Glitch.‘

Decision tools from parenting [§ Parenting]

  • Mental time travel — ‘how will you feel about this in a week / at forty?’ shrinks the present’s outsized feelings.
  • Nevertheless — feel-heard-but-still-decided, a leadership move as much as a parenting one.
  • Resulting, demonstrated: feeling like a great parent off the first child’s good behaviour, then learning it was temperament.

Connections