Boris Cherny with Diana Hu
Show: Y Combinator Startup School (Y Combinator)
Reformatted for readability — timestamps removed, lightly restructured. Not verbatim. For exact quotes, refer to the original transcript.
Host
All right, Boris, we're so excited to have you here, the creator of Claude Code. Thank you.
Boris Cherny
It's great to be here.
Host
Fresh off the press, you guys just shipped Opus V yesterday. Yes. And it seems that model performance keeps accelerating. You guys got took Arc AGI three to thirty percent, which is incredible.
Boris Cherny
Yes.
Host
And for context, before the best score was in in the low single digits or low teens, right? What can Opus V do now that it couldn't versus the previous version?
Boris Cherny
Yeah, there's um there's a lot that goes into every new model. And there's a lot of new capabilities that we teach and uh get the model to do. Whenever you do model training, you try to teach a whole bunch of different things, and most often it doesn't work. But some subset of the things the model does learn. And sometimes it also surprises you. It has these skills, it has abilities that you'd you actually didn't really teach it, but it it just kind of learned. For five, one example of something it does that I think no other model has done. Is it runs for a very long period of time. And especially when you combine Opus V with auto mode, it's just like incredible. Like it can go for days, weeks, months at a time. It just won't stop. You don't even need to use scaffolding. So you don't need splash goal, you don't need all this other stuff. It'll just go because it knows it needs to do the task. Um another thing that I'm really excited about, and um I'm gonna start, I think, to talk about a little bit more, um, but it's kinda surprising because it's such a new capability, is the model does not seem to be prompt injectable anymore.
Host
Not prompt injectable.
Boris Cherny
It's crazy. Like people have talked about this like lethal trifecta for a long time. And this really affects kind of harness design and agent design and product design. Because if the model reads some instruction on the internet that's like, you know, do X, and Y, and Z and also delete everything on the user's computer. A year ago the model would have just done it. But nowadays, Opus does not. And this has actually been the case since like Opus 4.7, 4.8. Uh Sonnet 5 has been quite good at this. Table was quite good at it. But Opus V just hits like a new frontier on this. So essentially, if you combine a well-aligned model, so this is like essentially three years of research into alignment, with a prompt injection classifier, which we run for all traffic. And what this is doing is it's based on Crystal's mechanistic interpretability work. Where it's it's literally we're looking at neurons in the model's brain that light up when prompt injection happens. So the model won't even tell you, but we can actually see those neurons and we can figure out and diagnose that it's happening. And then you combine that with the auto mode classifier. And with these three layers, we just cannot demonstrate prompt injection anymore.
Host
Talking about a prompt injection, the other side of the coin is now the system prompt. Let's talk a bit about the new release. You actually deleted over 80% of the system prompt from Claude Code. Yes. Tell us more about that.
Boris Cherny
I think something that a lot of people might not realize is Claude Code as a product and as a harness is just always changing. We're always adding stuff, we're always deleting stuff. Every time that a new model comes out, we delete a bunch of the system prompt, change a bunch of the system prompt, we change the set of tools all the time, we change the prompts for the tools all the time. And the reason is every model is very different. So something that you did for one model maybe three months ago, it just might not translate at all to the next model. And so one thing about Opus V is it's just really intelligent. And a lot of the stuff in the system prompt was correcting for these behaviors that the model should have known, but uh it didn't. Now Opus V just does it. So yeah, we deleted 80% of the system prompt. You can actually try deleting the rest of it too. Um so when you run Claude Code, you can just do like dash-system prompt and set whatever system prompt you want if you want to experiment with it. And another thing that you can try is um simple mode. So this is actually this kind of undocumented feature. If you do claude code simple equals one, like this uh environment variable, and then you run Claude, it'll delete all the system prompts, including from the tools. And we actually use this as a sort of ablation to figure out: is the prompt useful? And what's interesting is that the model is actually a little bit more intelligent without these prompts. That's something that we've been finding. But when you use Claude Code as a product, you do actually want some of these prompts because it helps you use the product and it it helps them the product behave and the model behave in the way that you would want when when you're using it as a person.
Host
I think the thing that's really fascinating in this era of building, basically you have to build the best harness in the world for. for Claude and that's Claude Code. From what I'm hearing, you for every model release, you basically delete all of the code base, delete all of the prompt and start from scratch every time. That in the old world would have been not something startup would have done for the product. It's like press delete every six months for everything.
Boris Cherny
That's right, that's right. We so to be fair, we don't delete the entire code base, but we do delete a lot. So every time there's a new model, we try we call in research you call this uh ablation. And so what this means is you delete the entire system prompt, and then you bring it back line by line to figure out what is the impact of each individual line. Um it's sort of like an eval, and you can kind of like evaluate it. And uh ablation essentially it's an eval, but you delete things to figure out the impact. And yeah, like we do the same thing for tools. Like we unship tools all the time. We, you know, delete code in the harness all the time. If you look at actually the code that's in the Claude Code harness today, almost all of it is about safety and permissions and static analysis. And there's a bunch of UI code. And we've actually unshipped a lot of the other code already.
Host
Do you think this way of building a agentic product and harness and basically doing ablations every time with a there's a new model release, should everyone in this room that's building AI products basically do that? Be comfortable and brave to press delete.
Boris Cherny
100%. Yeah, and and for people that aren't building agentic products but you're using Claude Code, every six months delete your Claude.md. Delete your skills.
Host
Let's talk a bit about how then you build this new prompt. When there's a new model release, like for everyone in the room, everyone will want to try Opus V and they're gonna press delete on their system prompt. How do they go about? Building rebuilding the system prompt. How do you set up your environment?
Boris Cherny
So you do it you do it kind of piece by piece. So the first step is you delete. The next step is you use it. And you don't wanna guess what's the instruction that the model needs because you might not predict it correctly. The thing that you wanna do is you wanna run it. And if it's like a custom agentic product that you're building, you wanna kind of run the product, uh, you wanna see where it fails with the model, you wanna see what it does well. If you're using Claude Code, you wanna see where it does well with your code base, or maybe where it stumbles over the architecture, or stumbles over something else. And only when you see it repeatedly stumble on the same thing, that's when you add it back. But you don't wanna do it too early. Because remember, like the model is gonna read this instruction every single time you use it. So you really wanna make sure that the model needs this instruction. I I think this is sort of the crazy thing about building on models, it's just so different than all the engineering that I've ever done. Like in the past, when you built on systems, you build these like big, beautiful systems, and you really think about the system design up front. You have like a big suite of unit tests, you think about everything, and you know, like a re-architecture is a big project. Sometimes it takes months. I've worked on re-architecture products at, you know, big companies that take years. And the motto is not like that. It's um the way to think about it is almost like a like a living creature, like as something more organic. It's a thing where every model generation it behaves differently. It has a slightly different personality. And you have to take the time to get to know it and then adjust the harness based on that. And I I think it's just very much like an empirical and kind of scientific thing. You have to take a very scientific mindset to it where you try something, you see the result, and then you iterate based on that.
Host
If you're building in this world right now, what then becomes uh stable? Are evals something that you keep from the previous models and keep using them in each new model release?
Boris Cherny
Um we do until we max out the eval.
Host
So that's sort of the tip for everyone. So code and system prompt, you have if you want to build at the bleeding edge and have the most capability for models, you gotta delete those. But evals are constant and keep appending to them, basically.
Boris Cherny
Yeah, you keep you keep appending. W what happens is um, you know, I actually wouldn't even go this far, to be honest. I think evals they outlive the harness a little bit, but not by that much. Like an eval might live for maybe one, two, three model generations. But nowadays the, you know, we're on the exponential. The model is improving so quickly. Very often we just saturate the eval and then we have to throw it away and we have to come up with a new eval. And this is just part of the process. And again, it's about being empirical. You have to use the product, you have to use the model, you have to see where it struggles, and then based on that, that's the evil set that you should build.
Host
I think one one term I heard you describe how to build the best agentic products on top of a Claude is this concept of uh unhobbling Claude. And tell us more more about what that means.
Boris Cherny
Yeah, so hobbling is this idea in a research that the model is doing something and you're just getting in the way. There there's this kind of like way of thinking about it that I really like. It's very useful when you're building product. And um it it it's called product overhang. And the idea is the model is able to do all sorts of things with today's models, not a future model but today's model, that we have not yet realized. And there are so many capabilities the model has like this that people are not aware of. And this is like the ability to maybe use a particular tool, use a particular language, solve a particular kind of problem, do things a particular kind of way that we thought was kind of beyond the model's capability. And there's this overhang because the model can do this at every given model generation. But there is often not a product that lets the model do this and lets it express this kind of ability to do this. And on the flip side, often what happens is the product gets in the way. And this getting in the way, we call this hobbling. And then not not eliciting the correct behavior from the model, we call this product overhang. So it's kind of like two sides of the same thing. What one example of this was the original Claude Code. When I first started working on it, this was um you know, like a year and a half, two years ago, something like that. This was like Summit three point five. At the time, that was an incredible coding model. That was like the best coding model that exists. Nowadays it's you know a pretty terrible coding model by modern standards. But that I think that was like the first great coding model that that we built as Anthropic. And at the time, if if you looked at the coding products of the time, what were they doing? They were doing like single-line autocomplete. They were doing sometimes multi-line autocomplete. That was sort of a new idea. They were doing chat. So you can talk to the agent, but it wasn't uh write access. You could only read. You could ask about the code base. And so the feeling was that there wasn't really a product. That was fully eliciting the model's capability to write entire functions at a time, entire files at a time. At the time it wasn't entire features, we weren't there yet, but probably entire files. That was the level of capability at the time. And so the idea with Claude Code was: all right, we think the model can probably do this, but if we get rid of all the scaffolding and just give the model the simplest possible harness so it can write an entire file at a time and build an entire feature.
Host
I think this is such a special insight for everyone here in the room. Basically, all of you could create the next Claude Code if you figure out how to unhobble the models, because that's effectively the birth story of Claude Code. You unhobble Sonnet 3.5 because all the previous iterations, we're still getting the model very rigid in in IDEs. And Claude Code was one of the first instances that gave it just a full terminal access.
Boris Cherny
Yes.
Host
And that then created this amazing product just that keeps going. So let's talk about um what are some areas on how should future founders here think about unhobbling Claude and fixing this product overhang.
Boris Cherny
So there's a couple of things that I will think about. One is you should give the model slightly harder tasks than what you think it can do. I think a a really common mistake that I see is people are using Claude Code, they're using quad, and they they just give it like way overly specific instructions. They're like, I want you to do this, but I want you to do it in this way, this way, this way. You must do like one, then two, then three, then four. And for modern models, that's actually really not the way to do it. You wanna go a little bit higher level. You wanna describe the task, you wanna describe the guardrails, you wanna describe like the exit criteria, and then just go with the model cook. And come back in a little bit. And I think it'll it'll surprise you. And again, like this is just not something that would have worked six months ago, but it does work today.
Host
Can you give some examples of these challenging tasks or capabilities that people should explore that it can do now, that it couldn't six months ago?
Boris Cherny
Yeah. So okay, one example is the model can now rewrite essentially any code base from one language to a different language. It's just sort of crazy. Like it's this work that would have taken just like a very long time as an engineer, and now the model's like quite fast at it. So so one example of this is um Claude Code is built on the Bun JavaScript runtime. It's a open source JavaScript runtime. Um it's an alternative to Node.js, it's kind of a faster node. Bun was written in Zig. Zig is a systems programming language. It's it's kinda like C, it's it's very low level. One of the problems with C with uh with Zig is you have to manually manage memory. And so it's quite easy to run into situations where there's like memory leaks and you know other memory management issues. And so one thing that the Bun team was doing is they were having Claude fuzz the code base and try to simulate and trigger memory weeks, and they were doing this for you know for a long period of time. They were able to find a lot of memory leaks, it was sort of like a case at a time. And that was kind of the capability of the model at the time, was doing this fuzzing. And then at some point, Jared on the team was like, Okay, let's just like rewrite it. Maybe the model can do this. And I I think this is like one of these test problems that he kind of threw at the model with every new model generation. And starting with Fable, the model started to be able to do it. And so I think Opus V could do it as well. And so what he did was essentially he defined a test suite. The nice thing about Bun is it's very, very well tested. There's a big test suite in Bun, there's a big test suite in Node.js, so it's easy to know if you did the right thing. And he had the model rewrite it from Zig to Rust. It was one prompt, it was a dynamic workflow. And dynamic workflows are a feature in Claude Code that essentially let you orchestrate, you know, dozens, hundreds, thousands of agents to do work productively. And it ran for eleven days and it rewrote the entire code base.
Host
And this was one shot.
Boris Cherny
It was one shot with well no, it wasn't one shot, but it there was steering. There was steering. Um but previous models just couldn't do this, even even with the steering. It just wouldn't have been possible.
Host
Just eleven days. Oh my god. This would have taken in the past, even with the best engineers, multiple months, years.
Boris Cherny
Over definitely over a year. Yeah. Yeah, over a year. This is like over a hundred thousand f like JavaScript runtime is really complicated. There's a there's a lot of stuff in there. Um and yeah, it like it works. This is in production now. This is what quad code uses now when when you're running it. So th this is kind of one example. I I would give a second example, so a product overhang. And so th this is like a practical use case where like there's a problem you're solving, it's like a business problem, an engineering problem, a product problem. And you should just keep throwing the latest model at it to see if it'll just do it. Because even if a previous model didn't, the new one might. I think the second way to think about it is experiment. And just give yourself like freedom to play with the model and do creative things. Often it'll surprise you. So something that's actually been really popular at uh internally that's been kind of viral within Anthropic the last couple weeks is someone figured out that you can give Opus V Open C D and you can have a draw. And so something you can do is you can ask Opus, like, hey, use OpenCV to like draw this image. And it's actually quite good. It can do like portraits, it can draw like animals, it can do like landscapes. And we didn't train the model to draw. Like it's just like the solicitation gap. Like if you ask it to do it the right way, it can just do it. And we discovered this kind of accidentally, just by playing around and trying creative things that didn't have direct commercial applications, um, but is just kind of interesting. And my hypothesis is there's probably dozens, hundreds of opportunities like this with the models of today that no one has yet realized.
Host
And the big area of research for this is basically model elicitation, right? Becoming really good at figuring out all these capabilities and asking the model to do the right thing, right?
Boris Cherny
Yes.
Host
How do people get better at that? And effectively, how do people get better at prompt engineering, do people still need to do a lot of prompt engineering, or is that changing as well? Tell us about where this is going.
Host
And as part of this, let's go deeper into this task that's still running two weeks since you launched it ago, two weeks ago. How many agents did it spawn?
Boris Cherny
No, I'm not sure. I'll I can ask quad and then I I can get back to you. I would I would guess thousands, tens of thousands?
Host
Has anyone in the audience had a uh prompt to to renew the models that run that spawn more than a thousand agents? Oh. I think this is another of the tips. Like the best cloud users are able to spawn tasks that are really providing you a lot of leverage, like thousands of agents.
Boris Cherny
Yes.
Host
How do you do that?
Boris Cherny
There's a few different ways to do it. The easiest way is dynamic workflows. To use dynamic workflows is a fairly new feature in Claude Code. And all you have to say is use a workflow. That's it. And then Claude will just trigger the dynamic workflow. What a dynamic workflow is, is essentially we have the bun we have the bun runtime, we use bun as a sandbox, and we start a virtual machine within Bun. And we let Claude start a lot of agents and orchestrate them. And it it doesn't just do one agent, it doesn't just do like 10 parallel agents. What it might do is um let's say a task is like rewrite the code base, or do really in-depth data analysis over some really complicated data, or maybe like build a very complex feature that takes multiple stages and maybe dozens of pull requests. And so what it's gonna do is it's it's gonna start a a bunch of agents to do kind of like the first pass. Based on that, it might do a second step where it has another set of agents that verify the work or that summarize the work. Then it might do like a third stage where it'll fan out again. So it'll kind of productively orchestrate a bunch of different agents. So my background is functional programming. And so the way that we design this is it's essentially an algebra for agents.
Host
I guess next conclusion from this, which you have mentioned in the past, that basically coding is solved, right? You have mentioned this. Um I'm curious now that effectively everyone can write software, what separates the exceptional builders? From the rest. What what are the qualities now that everyone can ship code?
Boris Cherny
So there's a way to run agents in sequence. There's a way to run agents in parallel. And Claude has different tools in order to orchestrate these agents inside of the sandbox to use tokens efficiently to do really, really complex work. It's kind of cool in something that just hasn't really been written about a lot. Like this is actually a new form of test-dem compute. Like when we talk about the scaling laws and kind of we talk about the model getting more intelligent over time. Historically, it's been a function of the size of the neural net, the amount of training data, and the number of flops that you put in to the training. And then recently we also added test M compute. So this is essentially a fancy way researcher way of saying how many tokens does it generate. And now dynamic workflows are essentially a new way to orchestrate test M compute. And it's a new way to kind of really, really ramp up the amount of test M compute that you use to do a really hard task. So this is all very long way to say, this is one way to launch thousands of agents in a way that is productive and efficient. A second way to do it is loops and routines. Loop is essentially a cron job that's running locally for quad. Routine is the same thing, but it's running in the quad, in the cloud. So you can close your laptop. And this is like slightly different because for a dynamic workflow, it's one task and you break it up into chunks. For loops and routines, it's one task that is repetitive, that doesn't share context, but it might share memory. And you kind of do this like over and over. You can do it like maybe every hour, every five minutes, every day. And so a thing that we've started doing is um we actually have quad maintaining itself now. And the way we do this is we have a Slack channel where we just had Claude start a bunch of different routines to maintain its own code base. And we actually do this for the CLI, for the iOS app, for the Android app, uh for the desktop app. And for example, one routine is clean up dead code. This is a single prompt, it's like one sentence. Quad runs this every day. It'll look for dead code across all the code bases using static and dynamic analysis. We didn't prompt that, it just kind of figured it out. And it'll put up pull requests every day to delete the dead code. Another example is uh shipping experiments that should go out. Um so the experiment's already out to 100%. It'll delete it from the code base and it'll just ship it. Another one is writing tests for areas of the code base that need test coverage. Another one is deleting tests that don't need to be there, because you know they were kind of useless tests added by older models or added by people at some point. One that one that I really love is this um I forget what we called it. I think we called it abstraction police. And the idea is there are often in a big code base, there's kind of the same abstraction and it appears multiple times. And if you kind of squint, it actually maybe should just be the same abstraction. But kind of over time, for whatever reason, you rebuilt it multiple ways in different parts of the code base. So quad kind of goes out every day across all our code bases, it finds these nearly duplicated abstractions and it unifies them. And so now we have every day maybe 20 or 30 of these routines. It's running across all of our code bases. And it's not totally there yet, but we're on the path to fully automating the maintenance of our apps by doing this. And this is again hundreds of agents running every day, sometimes thousands of agents every day. It's doing the work of dozens or hundreds of engineers. This is kind of what it used to take to do this kind of work. And this means that engineers can just like do the thing they actually want to do, which is ship new product and talk to users and uh do stuff that's actually fun.
Host
I guess next conclusion from this, which you have mentioned in the past, that basically coding is solved, right? You have mentioned this. Um I'm curious now that effectively everyone can write software, what separates the exceptional builders? From the rest. What what are the qualities now that everyone can ship code?
Boris Cherny
I I would give like one caveat. So coding is solved for the kind of coding that I do. It's not solved for everyone. You know, there's still code bases that are like super deep systems code bases where quad still struggles. There's distributed systems where quad still struggles. There's really kind of in the weeds UI verification, like something's off by Pixel or something. Quad is still not perfect at this. Like Opus V was a big leap in vision and computer use, but it's still not perfect. Um but I I'm actually curious, for people here, maybe raise your hand if a hundred percent of your code is written using agents. You don't write any code by hand anymore. It's pretty good. Okay, how about more than fifty percent? Slightly less hands, maybe about the same. Yeah. So I think it's like it's getting there. So it's kinda getting to this to, you know, to being solved for more and more kinds of code. And that's kind of cool. When I think about the people that are the best at using quad, I think there's a certain mindset that you can bring that's really effective. And it's really about being empirical. So forget all the things that you learned about past models. Forget everything that you've learned about computer science theory in class. Look at the model, try to do a task, see where it struggles, and then based on that, adjust. So it's just like very much become it's not a theoretical science, it's become an empirical science. So I think people that are really good at this, that are really good at kind of forgetting their priors, letting go of you know this like maybe idea that didn't work before, and just being open to trying it again. This is the kind of skill that's just very, very successful now.
Host
Now my last question is given everything that we talked about, if there's someone here that's studying CS and you you learned to program before this era of uh AI agency coding, what should students still learn the hard way, like the old way?
Boris Cherny
So for me, I learned computer science practically. I learned it by teaching myself to code in order to solve problems. Whenever I was doing this, I was doing it to solve a particular problem that I had. So I actually first learned to code on uh TI eighty three calculators. Um this is back in middle school. And um I ended up actually writing a guide on the internet for programming TI eighty three calculators. It's still up on the internet somewhere. Um and it was uh it was basic. That that was my first language. And I I learned how to program on calculators so I could just like get better at my math tests by uh by cheating on the test.
Host
So if I'm hearing and summarizing, start with making something you want first for yourself, and then level up and make something people want.
Boris Cherny
Yes.
Host
And we just have one last special announcement, Boris. You wanna one one last thing?
Boris Cherny
Yeah, so um for everyone here today, uh you are getting max twenty X. So look for look for a code in your email, and uh I can't wait to see what you build.
Host
We'll be sending you now. So I'm curious, someone in this room should be building something that runs hopefully multiple months and thousands of agents now that you have the account to do it. And with that, thank you so much, Boris.
Boris Cherny
Thank you.