Poteto on Verification, the Michelin Kitchen, and Review by Sampling

Poteto, creator of pstack, tells Matt Pocock how a month of about 2,500 merged pull requests became something he could stand behind. The lever is verification: the agent has to run and see its own work, or the human stays in the middle. The engineer’s job moves onto the environment, which he pictures as a Michelin kitchen rather than a factory. The volume then comes from an outer loop feeding coordinator agents, and from reviewing a sample instead of every pull request. Where the work cannot be checked programmatically, he says the method stops, and he does not have the answer. The video title places the work at SpaceX.

The 2,500 is Matt’s account of a talk Poteto posted on X about ten days earlier. Matt describes it as ‘how I shipped 2,500 PRs last month to production’, and says it ‘got about 3 million views or something’. In this conversation Poteto says ‘2,000 or however many’, ‘getting to 2,000 PRs’, and ‘2,000 and 2,500’. One month, and his own rounding of the same claim.

Key ideas

1. Verification is what makes a loop a loop, and the bottleneck moves to intent. The trust ladder is his name for giving agents harder and larger work only as trust grows, and the conversation is the story of climbing it. The climb starts before March. He had been on the React team at Meta, took a month off because he was burnt out, and started a side project. In January or February he was spending hours micromanaging one agent, writing skills, and unable to tell whether a skill had changed the result. That project became the basis of pstack, his toolkit of skills. He says the early skills are still open source. The aim was to extract how he works and give it to the agent.

   He says he joined Cursor in March. The work on Cursor’s agents window is early April. The window was laggy. He was reading flame graphs and heap snapshots by hand, and he calls himself the meat proxy between the agent and Chrome DevTools. The first skill that moved him up the ladder was verification, the first skill he built when he joined. Verification gives the agent hands and eyes: it runs the code, uses it as a person would, debugs, and takes traces and snapshots. Without that, the human is the proxy and there is no loop. Other skills still left him in the middle. If the agent cannot see the result, it cannot iterate. He says verification is the single most important skill in the toolkit even if you do not use pstack.

   ‘The most important part of a loop that allows it to be a loop is the verification part.’

   A loop, in his use, is an agent iterating on its own result. Hill climbing is the use he wanted on the agents window: a rubric or a score, and an agent that keeps trying to improve against it. The word is one he borrows from the labs. He points to a public autoresearch project by Andrej Karpathy as carrying the same idea. He says verification is probably the most important skill in pstack, that he spent a long time on it, and that each app he works on has a verification skill that maintains itself and that the team treats as critical infrastructure.

   The CLI inside the skill is glue, not a new kind of software. It interacts with a browser-automation tool and the Chrome DevTools protocol, and it calls a set of APIs. It encodes the deterministic steps so the agent does not rebuild them on every run and then throw them away. In January, or late the year before, the worry was the context window, because a summary left the session stupid. He still likes a clean window. The lasting reason for the CLI is speed, and that every agent then uses the same piece. A migration that is a mechanical rewrite of the syntax tree should be a script, not an agent. His principle is to let agents shine at the non-deterministic parts, because that is what they are trained to do, and to keep the mechanical parts deterministic. He treats determinism and judgment as a gradient. Mechanical steps become a script or a CLI. The agent keeps the steps that require thought.

   Domain expertise, he thinks, is more important than ever. As the models get better, the bottleneck is no longer the agent. It is saying the intent and the goal clearly enough that the agent can carry them out. A doctor or a lawyer who is a little tech-savvy, and who can articulate a product, is his example.

   ‘As the models get really, really, really good, it almost becomes like the bottleneck is no longer the agent, right? It becomes your ability to express your intent and your goals in a clear way that the agent can understand and actually carry out.’

   Matt’s prompt is that a word such as TDD, or a grilling session, can steer what the agent repeats in its trace. Poteto’s example is tautological tests, a tip of Matt’s that he says he copied. The word packs the constraint.

2. He wants a Michelin kitchen, and the new job is the environment. He thinks ‘software factory’ is an apt term, and he does not like it, because a factory does not suggest quality or craft. The kitchen does. He jokes that the name fits because he likes food. A home cook does every step. Filling the kitchen with family is, for most people, stressful. The question he cares about is how to add a few sous-chefs so the work is divided for the result, not for the sake of dividing. The chef stops cooking every plate and runs the kitchen: what to order, how to store it, when to prepare it. He calls that person the tech lead, and the chief executive of the kitchen. Engineers, as he sees it, are there now. They are not writing the code. They have agents. They are still responsible for the outcome, and their name is still on the work. Skills, the environment, and the codebase are the ingredients.

   The environment is the codebase, the lints, the types, the conventions, and the skills. His claim is that the engineer’s new job is to spend time here.

   ‘I almost feel like the new job of the engineer is really to spend time on the environment.’

   Matt’s definition of a good codebase, which Poteto takes up, is one that is easy to change because the paths are narrow. Lint and types are the start. The rest is one way to do each thing. Features each get a directory. A registry discovers them. The agent is not invited to append to a god file, one file that everything gets added to. The first versions of the desktop app were about eight god files, at least 10,000 lines each. The habit is to watch the agent fail, then turn the repeated mistake into a lint that makes the mistake impossible. He takes the picture from TypeScript narrowing, a wide type reduced to one constant, the subject of a talk he gave at a TypeScript conference years ago. He describes an internal framework for their Electron apps, not open source, built so there is really only one way.

   Below the trust ladder, the only option is to micromanage, and then there is no time to sharpen the tools. His analogy is a programmer who knows Notepad, has never heard of VS Code or Vim or Git, and is then under a deadline. A dull knife makes every meal slow. The same environment is what lets a new hire produce good code on the first day, not only an agent. He says he could not have reached the pull-request volume without that time on the kitchen.

3. The volume is loops and gardening, not 2,500 chats. He is not opening 2,500 chats. Once the kitchen exists he can run several projects at once and move between them, more present in some than in others. He splits an inner loop from an outer loop, and he says he may be using the words in his own way. The inner loop is the agents building towards a snapshot of his intent. The snapshot goes stale when bug reports, feature requests, and infrastructure limits arrive in Slack, Linear, or on X. For a long time he carried that outside context to the agent himself. Connecting the two loops is what removes him as the proxy. The agent can subscribe to a channel, triage, reproduce, and check that the bug is still on main rather than in the user’s setup.

   The theme he names is where he is the bottleneck, and how the agent answers its own question from real data instead of guessing. He thinks ‘company brain’ and ‘context graph’ are unnecessarily abstract. The concrete shape is a connector, with routines on Slack, X, email, and Linear, sending a bug in the desktop app to a Cursor project. A Cursor project, as he describes it, is a coordinator, or chief of staff: a cloud agent with its own computer that delegates and does not do the work. It can pick a topology, a way of splitting the work, which is what stops thirty related reports becoming thirty duplicated fixes. He can tell the connector to open a project without opening Cursor. Batching related reports, performance for example, is how the higher-level fault becomes visible. He offers a subscription to a Slack channel as the shape of the join.

   The pull requests are not all features. A lot of the work is gardening, his word for maintenance as against features, which is why 2,500 is not 2,500 features. An agent looks continuously for bad patterns in React, appends them to a document, and does not fix them first. Every couple of days he looks, and many rows are one pattern. The buffer forces the wider view that pure execution misses. A chief of staff can see the forest as well as run the work. He has the equivalent of more than ten chiefs of staff, on separate areas: performance of the desktop app, bugs users report, and a toy rewrite as a native app.

   The question he worked backwards from, and the unlock he names for the volume, is how the agent reaches the point of merging its own code. People who hear the number ask how anyone reviewed that many, and whether the team hates him. The answer is sampling, in the next idea.

4. Sample the food, correct the kitchen, and stop where the work cannot be verified. He does not want a state in which he never tastes the food, and he cannot taste every dish across several restaurants. Sampling is review by looking at some of the output and correcting the kitchen. The factory picture he will accept is the quality supervisor who samples. He looks at pull requests and at the code, and when many agents take the same shortcut he changes the skills, the constraints, the lints, or the types. He does not retrain that one agent. A one-off can stand. Done enough times, the environment guides the work and he can step back. He says this is hard, and he does not want it sold as something you get by using pstack.

   ‘You look at the code that the agents are writing and you scrutinize it very rigorously and you think about all the inefficiencies, the bad patterns that the agents are doing, and then you think about how to course-correct the environment, right? Not that single agent.’

   He accepts ‘dark factory’, Matt’s term, in one sense. If agents merge their own pull requests, the factory is dark: he sleeps, and the agents work. Full autopilot, a mode in pstack, runs several verifier agents, fuzzes the running app, fixes, and then lands the pull request. Fuzzing, as he defines it, means running the application and clicking through it as a person would, looking for regressions, and repeating until the pull request can land. It costs a lot of tokens. It can be turned down from something like ten verifiers to one, or to an instruction to verify. Confidence, for him, is verification plus the environment. He reviews after the merge, from the commit history in the morning, and if something is wrong he reverts, edits, or adds a lint. The first day he turned this on was scary, because something could break overnight. He says he sleeps better now.

   Matt then separates this from Vibe Coding, which he defines, following Karpathy, as the code almost not existing. Matt says Poteto’s method is the opposite: bad code and a bad environment produce bad output. Matt suggests a dimmer, some areas dark and some lit, and prefers the kitchen, because you still visit and taste.

   Matt’s last technical question is the one-way door. A two-way door, in Matt’s terms, can be merged and reverted. A one-way door cannot be cheaply undone. Medical, legal, and financial work are Matt’s examples, as are pull requests that lose data or cannot be walked back. Poteto says it comes back to how well the agent can verify. Where the domain is verifiable, a one-way door becomes a two-way door in a sense. Where the work is very hard to verify programmatically, it is very hard to get to the same point. Verifiability of the domain is the condition on letting an agent merge. Software is quite verifiable in a lot of cases, not totally. Some of mathematics is, if you write a proof. He says it is a great question and he does not really have the answer, and that engineers and the industry will have to figure it out. His hope is languages that join programs and proofs, so that a passing proof is a reason to merge. He names Lean and TLA+ as the older languages in which the proof lived apart from the program. The conditional is his, and he qualifies it at once: not every domain can be checked that way. He does not say that medical, legal, or financial work becomes a two-way door.

   ‘For domains where the work is verifiable, this is easier, and the one-way doors become two-way doors in a sense.’

5. A skill is a process written down, and the tools have to be your own. Matt defines a skill as a process turned into words, and asks how to combine pstack with his own library. Poteto’s method is one Matt had shared that day: read your old transcripts, find the prompts where you had to step in, and turn that correction into a skill so the agent stops repeating it. He says the two libraries are complementary. A skill is process, it is language, it is markdown. Nothing in the file is magical. People should carry their own knives. A chef takes their knives to the next restaurant. Trust, for him, is trust in those tools. He offers combinations as examples: Matt’s grill me, with docs, or wayfinder, together with execution skills in pstack. Some people will lean on one library more than the other. If you trust both of them, use both, and still mine your own chats. Past chats are a treasure trove because they are the process as it happened, not an idea of the process.

   The worked example is recall, a skill in pstack that pulls context out of earlier chats so he does not rewrite the prompt. He was fixing virtualisation bugs in the Cursor application. Each new chat needed the last chat. Recall collapses that retrieval. As the models improve, the old shape of a skill, the exact commands, can be deleted. What remains is the workflow, a series of steps, and skills get smaller. They can be cut apart and recombined because they are only language. Matt’s closing sentence is the host’s: if there is any magic, it is the words chosen and the thinking that turned a process into language.

Context

Who he is. He worked on the React team at Meta. He worked at Netflix, where managers talked about ‘context not control’ rather than micromanagement, and he thinks the phrase carries over to agents. He took a month off, started a side project, and joined Cursor in March. The video title places the shipping at SpaceX.

The earlier talk. About ten days before this livestream he posted a talk on X about shipping 2,500 pull requests in a month. Matt watched it, recommended it, and runs this conversation as a question session on it.

The previous livestream. Matt’s last conversation in this format was with Uncle Bob, on software quality and agents.

Related

• Poteto — guest
• Matt Pocock — host
• Vibe Coding — the mode Matt contrasts with this method, in which the code almost disappears
• Andrej Karpathy — the autoresearch mention, and the source of that vibe-coding definition
• Verifiability — his limit on self-merge, where a domain can be checked programmatically. Not that page’s account of reinforcement learning