Dario Amodei on Claude, AGI and the Future of AI

Dario Amodei

If you extrapolate the curves that we've had so far, right? If you say, "Well, I don't know, we're starting to get to PhD level, and last year we were at undergraduate level, and the year before we were at the level of a high school student," again, you can quibble with what tasks and for what. "We're still missing modalities, but those are being added," like computer use was added, like image generation has been added. If you just kind of eyeball the rate at which these capabilities are increasing, it does make you think that we'll get there by 2026 or 2027.

I think there are still worlds where it doesn't happen in 100 years. The number of those worlds is rapidly decreasing. We are rapidly running out of truly convincing blockers, truly compelling reasons why this will not happen in the next few years. The scale-up is very quick. We do this today, we make a model, and then we deploy thousands, maybe tens of thousands of instances of it. I think by the time, certainly within two to three years, whether we have these super powerful AIs or not, clusters are going to get to the size where you'll be able to deploy millions of these.

I am optimistic about meaning. I worry about economics and the concentration of power. That's actually what I worry about more, the abuse of power.

Dario Amodei

Yes, it's very frightening. It's very frightening.

Dario Amodei

So I can only describe it as it relates to my own experience, but I've been in the AI field for about 10 years and it was something I noticed very early on. So I first joined the AI world when I was working at Baidu with Andrew Ng in late 2014, which is almost exactly 10 years ago now. And the first thing we worked on, was speech recognition systems. And in those days I think deep learning was a new thing. It had made lots of progress, but everyone was always saying, "We don't have the algorithms we need to succeed. We are only matching a tiny fraction. There's so much we need to discover algorithmically. We haven't found the picture of how to match the human brain."

And in some ways it was fortunate, you can have almost beginner's luck. I was like a newcomer to the field. And I looked at the neural net that we were using for speech, the recurrent neural networks, and I said, "I don't know, what if you make them bigger and give them more layers? And what if you scale up the data along with this?" I just saw these as independent dials that you could turn. And I noticed that the models started to do better and better as you gave them more data, as you made the models larger, as you trained them for longer. And I didn't measure things precisely in those days, but along with colleagues, we very much got the informal sense that the more data and the more compute and the more training you put into these models, the better they perform.

And so initially my thinking was, "Hey, maybe that is just true for speech recognition systems. Maybe that's just one particular quirk, one particular area." I think it wasn't until 2017 when I first saw the results from GPT-1 that it clicked for me that language is probably the area in which we can do this. We can get trillions of words of language data, we can train on them. And the models we were trained in those days were tiny. You could train them on one to eight GPUs, whereas now we train jobs on tens of thousands, soon going to hundreds of thousands of GPUs.

And so when I saw those two things together, and there were a few people like Ilya Sudskever who you've interviewed, who had somewhat similar views. He might've been the first one, although I think a few people came to similar views around the same time, right? There was Rich Sutton's bitter lesson, Gwern wrote about the scaling hypothesis. But I think somewhere between 2014 and 2017 was when it really clicked for me, when I really got conviction that, "Hey, we're going to be able to these incredibly wide cognitive tasks if we just scale up the models."

And at every stage of scaling, there are always arguments. And when I first heard them honestly, I thought, "Probably I'm the one who's wrong and all these experts in the field are right. They know the situation better than I do, right?" There's the Chomsky argument about, "You can get syntactics but you can't get semantics." There was this idea, "Oh, you can make a sentence make sense, but you can't make a paragraph make sense." The latest one we have today is, "We're going to run out of data, or the data isn't high quality enough or models can't reason."

And each time, every time, we manage to either find a way around or scaling just is the way around. Sometimes it's one, sometimes it's the other. And so I'm now at this point, I still think it's always quite uncertain. We have nothing but inductive inference to tell us that the next two years are going to be like the last 10 years. But I've seen the movie enough times, I've seen the story happen for enough times to really believe that probably the scaling is going to continue, and that there's some magic to it that we haven't really explained on a theoretical basis yet.

Dario Amodei

Yes.

Dario Amodei

In particular, linear scaling up of bigger networks, bigger training times and more and more data. So all of these things, almost like a chemical reaction, you have three ingredients in the chemical reaction and you need to linearly scale up the three ingredients. If you scale up one, not the others, you run out of the other reagents and the reaction stops. But if you scale up everything in series, then the reaction can proceed.

Dario Amodei

Yeah, we've documented scaling laws in lots of domains other than language. So initially the paper we did that first showed it, was in early 2020, where we first showed it for language. There was then some work late in 2020 where we showed the same thing for other modalities like images, video, text to image, image to text, math. They all had the same pattern. And you're right, now there are other stages like post-training or there are new types of reasoning models. And in all of those cases that we've measured, we see similar types of scaling laws.

Dario Amodei

So in my previous career as a biophysicist… So I did a physics undergrad and then biophysics in grad school. So I think back to what I know as a physicist, which is actually much less than what some of my colleagues at Anthropic have in terms of expertise in physics. There's this concept called the one over F noise and one over X distributions, where often, just like if you add up a bunch of natural processes, you get a Gaussian, if you add up a bunch of differently-distributed natural processes… If you take a probe and hook it up to a resistor, the distribution of the thermal noise in the resistor goes as one over the frequency. It's some kind of natural convergent distribution.

And I think what it amounts to, is that if you look at a lot of things that are produced by some natural process that has a lot of different scales, not a Gaussian, which is kind of narrowly distributed, but if I look at large and small fluctuations that lead to electrical noise, they have this decaying one over X distribution. And so now I think of patterns in the physical world or in language. If I think about the patterns in language, there are some really simple patterns, some words are much more common than others, like the. Then there's basic noun-verb structure. Then there's the fact that nouns and verbs have to agree, they have to coordinate. And there's the higher-level sentence structure. Then there's the thematic structure of paragraphs. And so the fact that there's this regressing structure, you can imagine that as you make the networks larger, first they capture the really simple correlations, the really simple patterns, and there's this long tail of other patterns.

And if that long tail of other patterns is really smooth like it is with the one over F noise in physical processes like resistors, then you can imagine as you make the network larger, it's kind of capturing more and more of that distribution. And so that smoothness gets reflected in how well the models are at predicting and how well they perform.

Language is an evolved process. We've developed language, we have common words and less common words. We have common expressions and less common expressions. We have ideas, cliches, that are expressed frequently, and we have novel ideas. And that process has developed, has evolved with humans over millions of years. And so the guess, and this is pure speculation, would be that there's some kind of long tail distribution of the distribution of these ideas.

Dario Amodei

Exactly. If you have a small network, you only get the common stuff. If I take a tiny neural network, it's very good at understanding that a sentence has to have verb, adjective, noun, but it's terrible at deciding what those verb adjective and noun should be and whether they should make sense. If I make it just a little bigger, it gets good at that, then suddenly it's good at the sentences, but it's not good at the paragraphs. And so these rarer and more complex patterns get picked up as I add more capacity to the network.

Dario Amodei

Yeah.

Dario Amodei

I don't think any of us knows the answer to that question. My strong instinct would be that there's no ceiling below the level of humans. We humans are able to understand these various patterns. And so that makes me think that if we continue to scale up these models to kind of develop new methods for training them and scaling them up, that will at least get to the level that we've gotten to with humans. There's then a question of how much more is it possible to understand than humans do? How much is it possible to be smarter and more perceptive than humans? I would guess the answer has got to be domain-dependent.

If I look at an area like biology, and I wrote this essay, Machines of Loving Grace, it seems to me that humans are struggling to understand the complexity of biology. If you go to Stanford or to Harvard or to Berkeley, you have whole departments of folks trying to study the immune system or metabolic pathways, and each person understands only a tiny bit, a part of it, specializes. And they're struggling to combine their knowledge with that of other humans. And so I have an instinct that there's a lot of room at the top for AIs to get smarter.

If I think of something like materials in the physical world, or addressing conflicts between humans or something like that, I mean it may be there's only some of these problems are not intractable, but much harder. And it may be that there's only so well you can do at some of these things. Just like with speech recognition, there's only so clear I can hear your speech. So I think in some areas there may be ceilings that are very close to what humans have done. In other areas, those ceilings may be very far away. I think we'll only find out when we build these systems. It's very hard to know in advance. We can speculate, but we can't be sure.

Dario Amodei

Yes.

Dario Amodei

Yeah, I think in many cases, in theory, technology could change very fast. For example, all the things that we might invent with respect to biology, but remember, there's a clinical trial system that we have to go through to actually administer these things to humans. I think that's a mixture of things that are unnecessary in bureaucratic and things that kind of protect the integrity of society. And the whole challenge is that it's hard to tell what's going on. It's hard to tell which is which.

I think in terms of drug development, my view is that we're too slow and we're too conservative. But certainly if you get these things wrong, it's possible to risk people's lives by being too reckless. And so at least some of these human institutions are in fact protecting people. So it's all about finding the balance. I strongly suspect that balance is kind of more on the side of wishing to make things happen faster, but there is a balance.

Dario Amodei

So a few things, now we're talking about hitting the limit before we get to the level of humans and the skill of humans. So I think one that's popular today, and I think could be a limit that we run into, like most of the limits, I would bet against it, but it's definitely possible, is we simply run out of data. There's only so much data on the internet, and there's issues with the quality of the data. You can get hundreds of trillions of words on the internet, but a lot of it is repetitive or it's search engine optimization drivel, or maybe in the future it'll even be text generated by AIs itself. And so I think there are limits to what can be produced in this way.

That said, we, and I would guess other companies, are working on ways to make data synthetic, where you can use the model to generate more data of the type that you have already, or even generate data from scratch. If you think about what was done with DeepMind's AlphaGo Zero, they managed to get a bot all the way from no ability to play Go whatsoever to above human level, just by playing against itself. There was no example data from humans required in the AlphaGo Zero version of it.

The other direction of course, is these reasoning models that do chain of thought and stop to think and reflect on their own thinking. In a way that's another kind of synthetic data coupled with reinforcement learning. So my guess is with one of those methods, we'll get around the data limitation or there may be other sources of data that are available. We could just observe that, even if there's no problem with data, as we start to scale models up, they just stopped getting better. It seemed to be a reliable observation that they've gotten better, that could just stop at some point for a reason we don't understand.

The answer could be that we need to invent some new architecture. There have been problems in the past with say, numerical stability of models where it looked like things were leveling off, but actually when we found the right unblocker, they didn't end up doing so. So perhaps there's some new optimization method or some new technique we need to unblock things. I've seen no evidence of that so far, but if things were to slow down, that perhaps could be one reason.

Dario Amodei

So right now, I think most of the frontier model companies, I would guess, are operating in roughly 1 billion scale, plus or minus a factor of three. Those are the models that exist now or are being trained now. I think next year we're going to go to a few billion, and then 2026, we may go to above 10 billion. And probably by 2027, their ambitions to build hundred billion dollar clusters. And I think all of that actually will happen. There's a lot of determination to build the compute, to do it within this country, and I would guess that it actually does happen.

Now, if we get to a hundred billion, that's still not enough compute, that's still not enough scale, then either we need even more scale, or we need to develop some way of doing it more efficiently of shifting the curve. I think between all of these, one of the reasons I'm bullish about powerful AI happening so fast, is just that if you extrapolate the next few points on the curve, we're very quickly getting towards human level ability.

Some of the new models that we developed, some reasoning models that have come from other companies, they're starting to get to what I would call the PhD or professional level. If you look at their coding ability, the latest model we released, Sonnet 3.5, the new or updated version, it gets something like 50% on SWE-bench. And SWE-bench is an example of a bunch of professional real-world software engineering tasks. At the beginning of the year, I think the state of the art was 3 or 4%. So in 10 months we've gone from 3% to 50% on this task. And I think in another year we'll probably be at 90%. I mean, I don't know, but might even be less than that.

We've seen similar things in graduate-level math, physics, and biology from models like OpenAi's o1. So if we just continue to extrapolate this in terms of skill that we have, I think if we extrapolate the straight curve, within a few years, we will get to these models being above the highest professional level in terms of humans. Now, will that curve continue? You've pointed to, and I've pointed to a lot of possible reasons why that might not happen. But if the extrapolation curve continues, that is the trajectory we're on.

Dario Amodei

Yeah, so I want to separate out a couple things, right? Anthropic's mission is to kind of try to make this all go well. And we have a theory of change called Race to the Top. Race to the Top is about trying to push the other players to do the right thing by setting an example. It's not about being the good guy, it's about setting things up so that all of us can be the good guy.

I'll give a few examples of this. Early in the history of Anthropic, one of our co-founders, Chris Olah, who I believe you're interviewing soon, he's the co-founder of the field of mechanistic interpretability, which is an attempt to understand what's going on inside AI models. So we had him and one of our early teams focus on this area of interpretability, which we think is good for making models safe and transparent.

For three or four years that had no commercial application whatsoever. It still doesn't. Today we're doing some early betas with it, and probably it will eventually, but this is a very, very long research bed, and one in which we've built in public and shared our results publicly. And we did this because we think it's a way to make models safer. An interesting thing is that as we've done this, other companies have started doing it as well. In some cases because they've been inspired by it, in some cases because they're worried that if other companies are doing this, look more responsible, they want to look more responsible too. No one wants to look like the irresponsible actor. And so they adopt this as well. When folks come to Anthropic, interpretability is often a draw, and I tell them, "The other places you didn't go, tell them why you came here." And then you see soon that there's interpretability teams elsewhere as well.

And in a way that takes away our competitive advantage, because it's like, "Oh, now others are doing it as well." But it's good for the broader system, and so we have to invent some new thing that we're doing that others aren't doing as well. And the hope is to basically bid up the importance of doing the right thing. And it's not about us in particular. It's not about having one particular good guy. Other companies can do this as well. If they join the race to do this, that's the best news ever. It's about shaping the incentives to point upward instead of shaping the incentives to point downward.

Dario Amodei

Yes.

Dario Amodei

Trying to. I mean, I think we're still early in terms of our ability to see things, but I've been surprised at how much we've been able to look inside these systems and understand what we see. Unlike with the scaling laws where it feels like there's some law that's driving these models to perform better, on the inside, the models aren't… There's no reason why they should be designed for us to understand them, right? They're designed to operate, they're designed to work. Just like the human brain or human biochemistry. They're not designed for a human to open up the hatch, look inside and understand them. But we have found, and you can talk in much more detail about this to Chris, that when we open them up, when we do look inside them, we find things that are surprisingly interesting.

Dario Amodei

I'm amazed at how clean it's been. I'm amazed at things like induction heads. I'm amazed at things like that we can use sparse auto-encoders to find these directions within the networks, and that the directions correspond to these very clear concepts.

We demonstrated this a bit with the Golden Gate Bridge Claude. So this was an experiment where we found a direction inside one of the neural networks layers that corresponded to the Golden Gate Bridge. And we just turned that way up. And so we released this model as a demo, it was kind of half a joke, for a couple days, but it was illustrative of the method we developed. And you could take the model, you could ask it about anything. It would be like you could say, "How was your day?" And anything you asked, because this feature was activated, it would connect to the Golden Gate Bridge. So it would say, I'm feeling relaxed and expansive, much like the arches of the Golden Gate Bridge, or-

Dario Amodei

Somehow these interventions on the model, where you kind of adjust its behavior, somehow emotionally made it seem more human than any other version of the model.

Dario Amodei

It has a strong personality. It has these kind of obsessive interests. We can all think of someone who's obsessed with something. So it does make it feel somehow a bit more human.

Dario Amodei

Yeah, so let's go back to March when we first released these three models. So our thinking was different companies produce large and small models, better and worse models. We felt that there was demand, both for a really powerful model, and that might be a little bit slower that you'd have to pay more for, and also for fast cheap models that are as smart as they can be for how fast and cheap. Whenever you want to do some kind of difficult analysis, like if I want to write code for instance, or I want to brainstorm ideas or I want to do creative writing, I want the really powerful model.

But then there's a lot of practical applications in a business sense where it's like I'm interacting with a website, I am doing my taxes, or I'm talking to a legal advisor and I want to analyze a contract. Or we have plenty of companies that are just like, I want to do auto-complete on my IDE or something. And for all of those things, you want to act fast and you want to use the model very broadly. So we wanted to serve that whole spectrum of needs. So we ended up with this kind of poetry theme. And so what's a really short poem? It's a haiku. Haiku is the small, fast, cheap model that was at the time, was really surprisingly intelligent for how fast and cheap it was.

Sonnet is a medium-sized poem, write a couple paragraphs. And so Sonnet was the middle model. It is smarter but also a little bit slower, a little bit more expensive. And Opus, like a Magnum Opus is a large work, Opus was the largest, smartest model at the time. So that was the original kind of thinking behind it.

And our thinking then was, "Well, each new generation of models should shift that trade- off curve." So when we released Sonnet 3.5, it has roughly the same cost and speed as the Sonnet 3 model, but it increased its intelligence to the point where it was smarter than the original Opus 3 model. Especially for code, but also just in general. And so now we've shown results for Haiku 3.5. And I believe Haiku 3.5, the smallest new model, is about as good as Opus 3, the largest old model. So basically the aim here is to shift the curve and then at some point there's going to be an Opus 3.5.

Now every new generation of models has its own thing. They use new data, their personality changes in ways that we try to steer but are not fully able to steer. And so there's never quite that exact equivalence, where the only thing you're changing is intelligence. We always try and improve other things and some things change without us knowing or measuring. So it's very much an inexact science. In many ways, the manner and personality of these models is more an art than it is a science.

Dario Amodei

Yeah, so there's different processes. There's pre-training, which is just kind of the normal language model training. And that takes a very long time. That uses, these days, tens of thousands, sometimes many tens of thousands of GPUs or TPUs or training them, or we use different platforms, but accelerator chips, often training for months.

There's then a kind of post-training phase where we do reinforcement learning from human feedback as well as other kinds of reinforcement learning. That phase is getting larger and larger now, and often that's less of an exact science. It often takes effort to get it right. Models are then tested with some of our early partners to see how good they are, and they're then tested, both internally and externally, for their safety, particularly for catastrophic and autonomy risks. So we do internal testing according to our responsible scaling policy, which I could talk more about that in detail.

And then we have an agreement with the US and the UK AI Safety Institute, as well as other third-party testers in specific domains, to test the models for what are called CBRN risks, chemical, biological, radiological, and nuclear. We don't think that models pose these risks seriously yet, but every new model we want to evaluate to see if we're starting to get close to some of these more dangerous capabilities. So those are the phases, and then it just takes some time to get the model working in terms of inference and launching it in the API. So there's just a lot of steps to actually making a model work. And of course, we're always trying to make the processes as streamlined as possible.

We want our safety testing to be rigorous, but we want it to be rigorous and to be automatic, to happen as fast as it can, without compromising on rigor. Same with our pre-training process and our post-training process. So it's just building anything else. It's just like building airplanes. You want to make them safe, but you want to make the process streamlined. And I think the creative tension between those is an important thing in making the models work.

Dario Amodei

You would be surprised how much of the challenges of building these models comes down to software engineering, performance engineering. From the outside, you might think, "Oh man, we had this Eureka breakthrough." You know, this movie with the science. "We discovered it, we figured it out." But I think all things, even incredible discoveries, they almost always come down to the details. And often super, super boring details. I can't speak to whether we have better tooling than other companies. I mean, haven't been at those other companies, at least not recently, but it's certainly something we give a lot of attention to.

Dario Amodei

Yeah, I think at any given stage, we're focused on improving everything at once. Just naturally. Like, there are different teams. Each team makes progress in a particular area, in making their particular segment of the relay race better. And it's just natural that when we make a new model, we put all of these things in at once.

Dario Amodei

Yeah. Preference data from old models sometimes gets used for new models, although of course it performs somewhat better when it's trained on the new models. Note that we have this constitutional AI method such that we don't only use preference data, there's also a post-training process where we train the model against itself. And there's new types of post-training the model against itself that are used every day. So it's not just RLHF, a bunch of other methods as well. Post-training, I think, is becoming more and more sophisticated.

Dario Amodei

We observe that as well. By the way, there were a couple very strong engineers here at Anthropic, who all previous code models, both produced by us and produced by all the other companies, hadn't really been useful to them. They said, "Maybe this is useful to a beginner. It's not useful to me." But Sonnet 3.5, the original one for the first time, they said, "Oh, my God, this helped me with something that it would've taken me hours to do. This is the first model that's actually saved me time."

So again, the water line is rising. And then I think the new Sonnet has been even better. In terms of what it takes, I'll just say it's been across the board. It's in the pre-training, it's in the post-training, it's in various evaluations that we do. We've observed this as well. And if we go into the details of the benchmark, so SWE-bench is basically… Since you're a programmer, you'll be familiar with pull requests, and just pull requests, they're like a sort of atomic unit of work. You could say I'm implementing one thing.

So SWE-bench actually gives you a real world situation where the code base is in a current state and I'm trying to implement something that's described in language. We have internal benchmarks where we measure the same thing and you say, "Just give the model free rein to do anything, run anything, edit anything. How well is it able to complete these tasks?" And it's that benchmark that's gone from "it can do it 3% of the time" to "it can do it about 50% of the time."

So I actually do believe that you can gain benchmarks, but I think if we get to 100% on that benchmark in a way that isn't over-trained or game for that particular benchmark, probably represents a real and serious increase in programming ability. And I would suspect that if we can get to 90, 95% that it will represent ability to autonomously do a significant fraction of software engineering tasks.

Dario Amodei

Not giving you an exact date, but as far as we know, the plan is still to have a Claude 3.5 Opus.

Dario Amodei

Like Duke Nukem Forever?

Dario Amodei

What was that game? There was some game that was delayed 15 years.

Dario Amodei

Was that Duke Nukem Forever?

Dario Amodei

It's only been three months since we released the first Sonnet.

Dario Amodei

It just tells you about the pace, the expectations for when things are going to come out.

Dario Amodei

Naming is actually an interesting challenge here, right? Because I think a year ago, most of the model was pre-training. And so you could start from the beginning and just say, "Okay, we're going to have models of different sizes. We're going to train them all together and we'll have a family of naming schemes and then we'll put some new magic into them and then we'll have the next generation."

The trouble starts already when some of them take a lot longer than others to train. That already messes up your time a little bit. But as you make big improvement in pre-training, then you suddenly notice, "Oh, I can make better pre-train model." And that doesn't take very long to do, but clearly it has the same size and shape of previous models. So I think those two together as well as the timing issues. Any kind of scheme you come up with, the reality tends to frustrate that scheme, right? It tends to break out of the scheme.

It's not like software where you can say, "Oh, this is 3.7, this is 3.8." No, you have models with different trade-offs. You can change some things in your models, you can change other things. Some are faster and slower at inference. Some have to be more expensive, some have to be less expensive. And so I think all the companies have struggled with this. I think we were in a good position in terms of naming when we had Haiku, Sonnet and Opus.

Dario Amodei

We're trying to maintain it, but it's not perfect, so we'll try and get back to the simplicity. But just the nature of the field, I feel like no one's figured out naming. It's somehow a different paradigm from normal software and so none of the companies have been perfect at it. It's something we struggle with surprisingly much relative to how trivial it is for the grand science of training the models.

Dario Amodei

Yeah, yeah. I definitely think this question of there are lots of properties of the models that are not reflected in the benchmarks. I think that's definitely the case and everyone agrees. And not all of them are capabilities. Models can be polite or brusque, they can be very reactive or they can ask you questions. They can have what feels like a warm personality or a cold personality. They can be boring or they can be very distinctive like Golden Gate Claude was.

And we have a whole team focused on, I think we call it Claude character. Amanda leads that team and we'll talk to you about that, but it's still a very inexact science and often we find that models have properties that we're not aware of. The fact of the matter is that you can talk to a model 10,000 times and there are some behaviors you might not see just like with a human, right?

I can know someone for a few months and not know that they have a certain skill or not know that there's a certain side to them. And so I think we just have to get used to this idea. And we're always looking for better ways of testing our models to demonstrate these capabilities and also to decide which are the personality properties we want models to have and which we don't want to have. That itself, the normative question, is also super interesting.

Dario Amodei

From Reddit? Oh, boy.

Dario Amodei

So this actually doesn't apply. This isn't just about Claude. I believe I've seen these complaints for every foundation model produced by a major company. People said this about GPT-4, they said it about GPT-4 Turbo. So a couple things. One, the actual weights of the model, the actual brain of the model, that does not change unless we introduce a new model. There are just a number of reasons why it would not make sense practically to be randomly substituting in new versions of the model.

It's difficult from an inference perspective and it's actually hard to control all the consequences of changing the weights of the model. Let's say you wanted to fine-tune the model, I don't know, to say "certainly" less, which an old version of Sonnet used to do. You actually end up changing 100 things as well. So we have a whole process for it and we have a whole process for modifying the model. We do a bunch of testing on it. We do a bunch of user testing in early customers.

So we both have never changed the weights of the model without telling anyone. And certainly, in the current setup, it would not make sense to do that. Now, there are a couple things that we do occasionally do. One is sometimes we run A/B tests, but those are typically very close to when a model is being released and for a very small fraction of time.

So the day before the new Sonnet 3.5, I agree we should have had a better name. It's clunky to refer to it. There were some comments from people that it's gotten a lot better and that's because a fraction we're exposed to an A/B test for those one or two days. The other is that occasionally the system prompt will change. The system prompt can have some effects, although it's unlikely to dumb down models, it's unlikely to make them dumber.

And we've seen that while these two things, which I'm listing to be very complete, happened quite infrequently, the complaints for us and for other model companies about the model change, the model isn't good at this, the model got more censored, the model was dumbed down. Those complaints are constant and so I don't want to say people are imagining it or anything, but the models are, for the most part, not changing. If I were to offer a theory, I think it actually relates to one of the things I said before, which is that models are very complex and have many aspects to them. And so often, if I ask the model a question, if I'm like, "Do task X" versus, "Can you do task X?" the model might respond in different ways. And so there are all kinds of subtle things that you can change about the way you interact with the model that can give you very different results.

To be clear, this itself is like a failing by us and by the other model providers that the models are just often sensitive to small changes in wording. It's yet another way in which the science of how these models work is very poorly developed. And so if I go to sleep one night and I was talking to the model in a certain way and I slightly changed the phrasing of how I talk to the model, I could get different results.

So that's one possible way. The other thing is, man, it's just hard to quantify this stuff. It's hard to quantify this stuff. I think people are very excited by new models when they come out and then as time goes on, they become very aware of their limitations. So that may be another effect, but that's all a very long-winded way of saying for the most part, with some fairly narrow exceptions, the models are not changing.

Dario Amodei

It's amazing. Yeah.

Dario Amodei

And now I'm like, "I can't get this thing to work. This is such a piece of crap."

Dario Amodei

Yeah, so a couple points on this first. One is things that people say on Reddit and Twitter or X or whatever it is, there's actually a huge distribution shift between the stuff that people complain loudly about on social media and what actually statistically users care about and that drives people to use the models.

People are frustrated with things like the model not writing out all the code or the model just not being as good at code as it could be, even though it's the best model in the world on code. I think the majority of things are about that, but certainly a vocal minority raise these concerns, are frustrated by the model refusing things that it shouldn't refuse or apologizing too much or just having these annoying verbal tics.

The second caveat, and I just want to say this super clearly because I think some people don't know it, others know it, but forget it. It is very difficult to control across the board how the models behave. You cannot just reach in there and say, "Oh, I want the model to apologize less." You can do that. You can include training data that says, "Oh, the model should apologize less." But then in some other situation, they end up being super rude or overconfident in a way that's misleading people.

So there are all these trade-offs. For example, another thing is if there was a period during which models, ours and I think others as well, were too verbose, they would repeat themselves, they would say too much. You can cut down on the verbosity by penalizing the models for just talking for too long. What happens when you do that, if you do it in a crude way, is when the models are coding, sometimes they'll say, "Rest of the code goes here," right?

Because they've learned that that's the way to economize and that they see it. And then so that leads the model to be so-called lazy in coding where they're just like, "Ah, you can finish the rest of it." It's not because we want to save on compute or because the models are lazy during winter break or any of the other conspiracy theories that have come up. Actually, it's just very hard to control the behavior of the model, to steer the behavior of the model in all circumstances at once.

There's this whack- a-mole aspect where you push on one thing and these other things start to move as well that you may not even notice or measure. And so one of the reasons that I care so much about grand alignment of these AI systems in the future is actually, these systems are actually quite unpredictable. They're actually quite hard to steer and control. And this version we're seeing today of you make one thing better, it makes another thing worse, I think that's like a present day analog of future control problems in AI systems that we can start to study today.

I think that difficulty in steering the behavior and making sure that if we push an AI system in one direction, it doesn't push it in another direction in some other ways that we didn't want. I think that's an early sign of things to come, and if we can do a good job of solving this problem of you ask the model to make and distribute smallpox and it says no, but it's willing to help you in your graduate level virology class, how do we get both of those things at once? It's hard.

It's very easy to go to one side or the other and it's a multidimensional problem. And so I think these questions of shaping the model's personality, I think they're very hard. I think we haven't done perfectly on them. I think we've actually done the best of all the AI companies, but still so far from perfect.

And I think if we can get this right, if we can control the false positives and false negatives in this very controlled present day environment, we'll be much better at doing it for the future when our worry is: will the models be super autonomous? Will they be able to make very dangerous things? Will they be able to autonomously build whole companies and are those companies aligned? So I think of this present task as both vexing but also good practice for the future.

Dario Amodei

So typically, we'll have internal model bashings where all of Anthropic… Anthropic is almost 1,000 people. People just try and break the model. They try and interact with it various ways. We have a suite of evals for, "Oh, is the model refusing in ways that it couldn't?" I think we even had a "certainly" eval because again, at one point, the model had this problem where it had this annoying tick where it would respond to a wide range of questions by saying, "Certainly, I can help you with that. Certainly, I would be happy to do that. Certainly, this is correct."

And so we had a "certainly" eval, which is: how often does the model say certainly? But look, this is just a whack-a-mole. What if it switches from "certainly" to "definitely"? So every time we add a new eval and we're always evaluating for all the old things, we have hundreds of these evaluations, but we find that there's no substitute for a human interacting with it.

And so it's very much like the ordinary product development process. We have hundreds of people within Anthropic bash the model. Then we do external A/B tests. Sometimes we'll run tests with contractors. We pay contractors to interact with the model. So you put all of these things together and it's still not perfect. You still see behaviors that you don't quite want to see. You still see the model refusing things that it just doesn't make sense to refuse.

But I think trying to solve this challenge, trying to stop the model from doing genuinely bad things that everyone agrees it shouldn't do, everyone agrees that the model shouldn't talk about, I don't know, child abuse material. Everyone agrees the model shouldn't do that, but at the same time, that it doesn't refuse in these dumb and stupid ways.

I think drawing that line as finely as possible, approaching perfectly, is still a challenge and we're getting better at it every day, but there's a lot to be solved. And again, I would point to that as an indicator of a challenge ahead in terms of steering much more powerful models.

Dario Amodei

I don't want to commit to any naming scheme because if I say here, "We're going to have Claude 4 next year," and then we decide that we should start over because there's a new type of model, I don't want to commit to it. I would expect in a normal course of business that Claude 4 would come after Claude 3. 5, but you never know in this wacky field.

Dario Amodei

Scaling is continuing. There will definitely be more powerful models coming from us than the models that exist today. That is certain. Or if there aren't, we've deeply failed as a company.

Dario Amodei

As much as I am excited about the benefits of these models, and we'll talk about that if we talk about Machines of Loving Grace, I'm worried about the risks and I continue to be worried about the risks. No one should think that Machines of Loving Grace was me saying I'm no longer worried about the risks of these models. I think they're two sides of the same coin.

The power of the models and their ability to solve all these problems in biology, neuroscience, economic development, governance and peace, large parts of the economy, those come with risks as well, right? With great power comes great responsibility. The two are paired. Things that are powerful can do good things and they can do bad things. I think of those risks as being in several different categories, perhaps the two biggest risks that I think about. And that's not to say that there aren't risks today that are important, but when I think of really the things that would happen on the grandest scale, one is what I call catastrophic misuse.

These are misuse of the models in domains like cyber, bio, radiological, nuclear, things that could harm or even kill thousands, even millions of people if they really, really go wrong. These are the number one priority to prevent. And here I would just make a simple observation, which is that the models, if I look today at people who have done really bad things in the world, I think actually humanity has been protected by the fact that the overlap between really smart, well-educated people and people who want to do really horrific things has generally been small.

Let's say I'm someone who I have a PhD in this field, I have a well-paying job. There's so much to lose. Even assuming I'm completely evil, which most people are not, why would such a person risk their life, risk their legacy, their reputation to do something truly, truly evil? If we had a lot more people like that, the world would be a much more dangerous place. And so my worry is that by being a much more intelligent agent, AI could break that correlation.

And so I do have serious worries about that. I believe we can prevent those worries. But I think as a counterpoint to Machines of Loving Grace, I want to say that there's still serious risks. And the second range of risks would be the autonomy risks, which is the idea that models might, on their own, particularly as we give them more agency than they've had in the past, particularly as we give them supervision over wider tasks like writing whole code bases or someday even effectively operating entire companies, they're on a long enough leash. Are they doing what we really want them to do?

It's very difficult to even understand in detail what they're doing, let alone control it. And like I said, these early signs that it's hard to perfectly draw the boundary between things the model should do and things the model shouldn't do that if you go to one side, you get things that are annoying and useless and you go to the other side, you get other behaviors. If you fix one thing, it creates other problems.

We're getting better and better at solving this. I don't think this is an unsolvable problem. I think this is a science like the safety of airplanes or the safety of cars or the safety of drugs. I don't think there's any big thing we're missing. I just think we need to get better at controlling these models. And so these are the two risks I'm worried about. And our responsible scaling plan, which I'll recognize is a very long-winded answer to your question.

Dario Amodei

Our responsible scaling plan is designed to address these two types of risks. And so every time we develop a new model, we basically test it for its ability to do both of these bad things. So if I were to back up a little bit, I think we have an interesting dilemma with AI systems where they're not yet powerful enough to present these catastrophes. I don't know if they'll ever present these catastrophes. It's possible they won't.

But the case for worry, the case for risk is strong enough that we should act now and they're getting better very, very fast. I testified in the Senate that we might have serious bio risks within two to three years. That was about a year ago. Things have proceeded apace. So we have this thing where it's surprisingly hard to address these risks because they're not here today, they don't exist. They're like ghosts, but they're coming at us so fast because the models are improving so fast.

So how do you deal with something that's not here today, doesn't exist, but is coming at us very fast? So the solution we came up with for that, in collaboration with people like the organization METR and Paul Christiano is what you need for that are you need tests to tell you when the risk is getting close. You need an early warning system. And so every time we have a new model, we test it for its capability to do these CBRN tasks as well as testing it for how capable it is of doing tasks autonomously on its own.

And in the latest version of our RSP, which we released in the last month or two, the way we test autonomy risks is the AI model's ability to do aspects of AI research itself, which when the AI models can do AI research, they become truly, truly autonomous. And that threshold is important for a bunch of other ways. And so what do we then do with these tasks? The RSP basically develops what we've called an if-then structure, which is if the models pass a certain capability, then we impose a certain set of safety and security requirements on them.

So today's models are what's called ASL-2. Models that were ASL-1 is for systems that manifestly don't pose any risk of autonomy or misuse. So for example, a chess playing bot, Deep Blue would be ASL-1. It's just manifestly the case that you can't use Deep Blue for anything other than chess. It was just designed for chess. No one's going to use it to conduct a masterful cyber attack or to run wild and take over the world.

ASL-2 is today's AI systems where we've measured them and we think these systems are simply not smart enough to autonomously self-replicate or conduct a bunch of tasks and also not smart enough to provide meaningful information about CBRN risks and how to build CBRN weapons above and beyond what can be known from looking at Google. In fact, sometimes they do provide information above and beyond a search engine, but not in a way that can be stitched together, not in a way that end-to-end is dangerous enough.

So ASL-3 is going to be the point at which the models are helpful enough to enhance the capabilities of non-state actors, right? State actors can already do, unfortunately, to a high level of proficiency, a lot of these very dangerous and destructive things. The difference is that non-state actors are not capable of it. And so when we get to ASL-3, we'll take special security precautions designed to be sufficient to prevent theft of the model by non-state actors and misuse of the model as it's deployed. We'll have to have enhanced filters targeted at these particular areas.

Dario Amodei

Cyber, bio, nuclear and model autonomy, which is less a misuse risk and more a risk of the model doing bad things itself. ASL-4, getting to the point where these models could enhance the capability of a already knowledgeable state actor and/or become the main source of such a risk. If you wanted to engage in such a risk, the main way you would do it is through a model. And then I think ASL-4 on the autonomy side, it's some amount of acceleration in AI research capabilities with an AI model.

And then ASL-5 is where we would get to the models that are truly capable that it could exceed humanity in their ability to do any of these tasks. And so the point of the if-then structure commitment is basically to say, "Look, I don't know, I've been working with these models for many years and I've been worried about risk for many years. It's actually dangerous to cry wolf. It's actually dangerous to say this model is risky. And people look at it and they say this is manifestly not dangerous." Again, it's the delicacy of the risk isn't here today, but it's coming at us fast.

How do you deal with that? It's really vexing to a risk planner to deal with it. And so this if-then structure basically says, "Look, we don't want to antagonize a bunch of people, we don't want to harm our own ability to have a place in the conversation by imposing these very onerous burdens on models that are not dangerous today." So the if-then, the trigger commitment is basically a way to deal with this. It says you clamp down hard when you can show the model is dangerous.

And of course, what has to come with that is enough of a buffer threshold that you're not at high risk of missing the danger. It's not a perfect framework. We've had to change it. We came out with a new one just a few weeks ago and probably going forward, we might release new ones multiple times a year because it's hard to get these policies right technically, organizationally from a research perspective. But that is the proposal, if-then commitments and triggers in order to minimize burdens and false alarms now, but really react appropriately when the dangers are here.

Dario Amodei

Yeah. So that is hotly debated within the company. We are working actively to prepare ASL-3 security measures as well as ASL-3 deployment measures. I'm not going to go into detail, but we've made a lot of progress on both and we're prepared to be, I think, ready quite soon. I would not be surprised at all if we hit ASL-3 next year. There was some concern that we might even hit it this year. That's still possible. That could still happen. It's very hard to say, but I would be very, very surprised if it was 2030. I think it's much sooner than that.

Dario Amodei

Yes.

Dario Amodei

Yeah. I think for ASL-3, it's primarily about security and about filters on the model relating to a very narrow set of areas when we deploy the model. Because at ASL-3, the model isn't autonomous yet. And so you don't have to worry about the model itself behaving in a bad way even when it's deployed internally. So I think the ASL- 3 measures are, I won't say straightforward, they're rigorous, but they're easier to reason about.

I think once we get to ASL-4, we start to have worries about the models being smart enough that they might sandbag tests, they might not tell the truth about tests. We had some results came out about sleeper agents and there was a more recent paper about, "Can the models mislead attempts to sandbag their own abilities, present themselves as being less capable than they are?" And so I think with ASL-4, there's going to be an important component of using other things than just interacting with the models.

For example, interpretability or hidden chains of thought where you have to look inside the model and verify via some other mechanism that is not as easily corrupted as what the model says, that the model indeed has some property. So we're still working on ASL-4. One of the properties of the RSP is that we don't specify ASL-4 until we've hit ASL-3. And I think that's proven to be a wise decision because even with ASL-3, again, it's hard to know this stuff in detail, and we want to take as much time as we can possibly take to get these things right.

Dario Amodei

Humans, yes.

Dario Amodei

For ASL- 4, it's both, I think.

Dario Amodei

Yeah. Of course, you can hook up the mechanistic interpretability to the model itself, but then you've lost it as a reliable indicator of the model state. There are a bunch of exotic ways you can think of that it might also not be reliable, like if the model gets smart enough that it can jump computers and read the code where you're looking at its internal state. We've thought about some of those. I think they're exotic enough. There are ways to render them unlikely. But yeah, generally, you want to preserve mechanistic interpretability as a verification set or test set that's separate from the training process of the model.

Dario Amodei

Oh, yeah. Yeah. We've seen lots of examples of demagoguery in our life from humans, and there's a concern that models could do that as well.

Dario Amodei

Yeah. It's actually relatively simple. So Claude has had for a long time, since Claude 3 back in March, the ability to analyze images and respond to them with text. The only new thing we added is those images can be screenshots of a computer and in response, we train the model to give a location on the screen where you can click and/or buttons on the keyboard, you can press in order to take action. And it turns out that with actually not all that much additional training, the models can get quite good at that task. It's a good example of generalization. People sometimes say if you get to lower earth orbit, you're halfway to anywhere because of how much it takes to escape the gravity well. If you have a strong pre-trained model, I feel like you're halfway to anywhere in terms of the intelligence space. And so actually, it didn't take all that much to get Claude to do this. And you can just set that in a loop, give the model a screenshot, tell it what to click on, give it the next screenshot, tell it what to click on and that turns into a full kind of almost 3D video interaction of the model and it's able to do all of these tasks. We showed these demos where it's able to fill out spreadsheets, it's able to kind of interact with a website, it's able to open all kinds of programs, different operating systems, Windows, Linux, Mac. So I think all of that is very exciting. I will say, while in theory there's nothing you could do there that you couldn't have done through just giving the model the API to drive the computer screen, this really lowers the barrier. And there's a lot of folks who either aren't in a position to interact with those APIs or it takes them a long time to do.

It's just the screen is just a universal interface that's a lot easier to interact with. And so I expect over time, this is going to lower a bunch of barriers. Now, honestly, the current model has, it leaves a lot still to be desired and we were honest about that in the blog. It makes mistakes, it misclicks. We were careful to warn people, "Hey, you can't just leave this thing to run on your computer for minutes and minutes. You got to give this thing boundaries and guardrails." And I think that's one of the reasons we released it first in an API form rather than just hand the consumer and give it control of their computer. But I definitely feel that it's important to get these capabilities out there. As models get more powerful, we're going to have to grapple with how do we use these capabilities safely. How do we prevent them from being abused?

And I think releasing the model while the capabilities are still limited is very helpful in terms of doing that. I think since it's been released, a number of customers, I think Replit was maybe one of the most quickest to deploy things, have made use of it in various ways. People have hooked up demos for Windows desktops, Macs, Linux machines. So yeah, it's been very exciting. I think as with anything else, it comes with new exciting abilities and then with those new exciting abilities, we have to think about how to make the model safe, reliable, do what humans want them to do. It's the same story for everything. Same thing. It's that same tension.

Dario Amodei

Yeah. I think speaking at a high level, it's our intention to keep investing a lot in making the model better. I think we look at some of the benchmarks where previous models were like, "Oh, could do it 6% of the time," and now our model would do it 14 or 22% of the time. And yeah, we want to get up to the human level reliability of 80, 90% just like anywhere else. We're on the same curve that we were on with SWE-bench where I think I would guess a year from now, the models can do this very, very reliably. But you got to start somewhere.

Dario Amodei

It depends what you mean by special and special in general, but I generally think the same kinds of techniques that we've been using to train the current model, I expect that doubling down on those techniques in the same way that we have for code, for models in general, for image input, for voice, I expect those same techniques will scale here as they have everywhere else,

Dario Amodei

Yeah, yeah. No and we've been very aware of that. Look, my view actually is computer use isn't a fundamentally new capability like the CBRN or autonomy capabilities are. It's more like it kind of opens the aperture for the model to use and apply its existing abilities. And so the way we think about it, going back to our RSP, is nothing that this model is doing inherently increases the risk from an RSP perspective, but as the models get more powerful, having this capability may make it scarier once it has the cognitive capability to do something at the ASL-3 and ASL-4 level, this may be the thing that kind of unbounds it from doing so. So going forward, certainly this modality of interaction is something we have tested for and that we will continue to test for an RSP going forward. I think it's probably better to learn and explore this capability before the model is super capable

Dario Amodei

Yeah, we've thought a lot about things like spam, CAPTCHA, mass… One secret, I'll tell you, if you've invented a new technology, not necessarily the biggest misuse, but the first misuse you'll see, scams, just petty scams.

Dario Amodei

It's like a thing as old, people scamming each other, it's this thing as old as time. And it's just every time, you got to deal with it.

Dario Amodei

Yeah, yeah.

Dario Amodei

Like I said, there are a lot of petty criminals in the world and it's like every new technology is a new way for petty criminals to do something stupid and malicious.

Dario Amodei

Yeah, we sandbox during training. So for example, during training we didn't expose the model to the internet. I think that's probably a bad idea during training because the model can be changing its policy, it can be changing what it's doing and it's having an effect in the real world. In terms of actually deploying the model, it kind of depends on the application. Sometimes you want the model to do something in the real world. But of course, you can always put guard, you can always put guard rails on the outside. You can say, "Okay, well, this model's not going to move data from my, the model's not going to move any files from my computer or my web server to anywhere else."

Now, when you talk about sandboxing, again, when we get to ASL-4, none of these precautions are going to make sense there. When you talk about ASL-4, you're then, the model is being, there's theoretical worry the model could be smart enough to kind of break it to out of any box. And so there, we need to think about mechanistic interpretability. If we're going to have a sandbox, it would need to be a mathematically provable. That's a whole different world than what we're dealing with with the models today.

Dario Amodei

I think it's probably not the right approach. I think the right approach, instead of having something unaligned that you're trying to prevent it from escaping, I think it's better to just design the model the right way or have a loop where you look inside the model and you're able to verify properties and that gives you an opportunity to tell, iterate and actually get it right. I think containing bad models is a much worse solution than having good models.

Dario Amodei

Yes, we ended up making some suggestions to the bill. And then some of those were adopted and we felt, I think, quite positively about the bill by the end of that, it did still have some downsides. And of course, it got vetoed. I think at a high level, I think some of the key ideas behind the bill are I would say similar to ideas behind our RSPs. And I think it's very important that some jurisdiction, whether it's California or the federal government and/or other countries and other states, passes some regulation like this. And I can talk through why I think that's so important. So I feel good about our RSP. It's not perfect. It needs to be iterated on a lot. But it's been a good forcing function for getting the company to take these risks seriously, to put them into product planning, to really make them a central part of work at Anthropic and to make sure that all of a thousand people, and it's almost a thousand people now at Anthropic, understand that this is one of the highest priorities of the company, if not the highest priority.

But one, there are still some companies that don't have RSP like mechanisms, like OpenAI, Google did adopt these mechanisms a couple months after Anthropic did, but there are other companies out there that don't have these mechanisms at all. And so if some companies adopt these mechanisms and others don't, it's really going to create a situation where some of these dangers have the property that it doesn't matter if three out of five of the companies are being safe, if the other two are being unsafe, it creates this negative externality. And I think the lack of uniformity is not fair to those of us who have put a lot of effort into being very thoughtful about these procedures. The second thing is I don't think you can trust these companies to adhere to these voluntary plans on their own. Right? I like to think that Anthropic will, we do everything we can that we will, our RSP is checked by our long-term benefit trust, so we do everything we can to adhere to our own RSP.

But you hear lots of things about various companies saying, "Oh, they said they would give this much compute and they didn't. They said they would do this thing and the didn't." I don't think it makes sense to litigate particular things that companies have done, but I think this broad principle that if there's nothing watching over them, if there's nothing watching over us as an industry, there's no guarantee that we'll do the right thing and the stakes are very high. And so I think it's important to have a uniform standard that everyone follows and to make sure that simply that the industry does what a majority of the industry has already said is important and has already said that they definitely will do.

Right, some people, I think there's a class of people who are against regulation on principle. I understand where that comes from. If you go to Europe and you see something like GDPR, you see some of the other stuff that they've done. Some of it's good, but some of it is really unnecessarily burdensome and I think it's fair to say really has slowed innovation. And so I understand where people are coming from on priors. I understand why people start from that position. But again, I think AI is different. If we go to the very serious risks of autonomy and misuse that I talked about just a few minutes ago, I think that those are unusual and they warrant an unusually strong response. And so I think it's very important.

Again, we need something that everyone can get behind. I think one of the issues with SB 1047, especially the original version of it was it had a bunch of the structure of RSPs, but it also had a bunch of stuff that was either clunky or that just would've created a bunch of burdens, a bunch of hassle and might even have missed the target in terms of addressing the risks. You don't really hear about it on Twitter, you just hear about kind of people are cheering for any regulation. And then the folks who are against make up these often quite intellectually dishonest arguments about how it'll make us move away from California, bill doesn't apply if you're headquartered in California, bill only applies if you do business in California, or that it would damage the open source ecosystem or that it would cause all of these things.

I think those were mostly nonsense, but there are better arguments against regulation. There's one guy, Dean Ball, who's really, I think, a very scholarly analyst who looks at what happens when a regulation is put in place in ways that they can kind of get a life of their own or how they can be poorly designed. And so our interest has always been we do think there should be regulation in this space, but we want to be an actor who makes sure that that regulation is something that's surgical, that's targeted at the serious risks and is something people can actually comply with. Because something I think the advocates of regulation don't understand as well as they could is if we get something in place that's poorly targeted, that wastes a bunch of people's time, what's going to happen is people are going to say, "See, these safety risks, this is nonsense. I just had to hire 10 lawyers to fill out all these forms. I had to run all these tests for something that was clearly not dangerous."

And after six months of that, there will be a ground swell and we'll end up with a durable consensus against regulation. And so I think the worst enemy of those who want real accountability is badly designed regulation. We need to actually get it right. And if there's one thing I could say to the advocates, it would be that I want them to understand this dynamic better and we need to be really careful and we need to talk to people who actually have experience seeing how regulations play out in practice. And the people who have seen that, understand to be very careful. If this was some lesser issue, I might be against regulation at all.

But what I want the opponents to understand is that the underlying issues are actually serious. They're not something that I or the other companies are just making up because of regulatory capture, they're not sci-fi fantasies, they're not any of these things. Every time we have a new model, every few months we measure the behavior of these models and they're getting better and better at these concerning tasks just as they are getting better and better at good, valuable, economically useful tasks. And so I would just love it if some of the former, I think SB 1047 was very polarizing, I would love it if some of the most reasonable opponents and some of the most reasonable proponents would sit down together. And I think that the different AI companies, Anthropic was the only AI company that felt positively in a very detailed way. I think Elon tweeted briefly something positive, but some of the big ones like Google, OpenAI, Meta, Microsoft were pretty staunchly against.

---

Note

This transcript has been cleaned of all timestamp links and condensed from the original content provided. The full transcript extends beyond the excerpt provided, covering additional chapters on hiring teams, post-training, constitutional AI, AGI timelines, and discussions with Amanda Askell and Chris Olah.