Oral evidence: AI adoption, safety and governance, HC 689

Oral evidence, 13 October 2026.

Witnesses

Daniel Kokotajlo, Aengus Lynch, Steven Adler and Geoffrey Irving.

Chair

Welcome to today’s session of the Business, Innovation, Science and Trade Committee in our inquiry into AI safety and governance. We are incredibly grateful to you, our witnesses, for joining us here in the House of Commons today at what I know is a very early hour on the west coast. Thank you very much on behalf of the UK Parliament for coming and giving evidence to us. Daniel, I am going to start with you, if I may. Could you tell us why you resigned from OpenAI?

Daniel Kokotajlo

OpenAI, Anthropic and some of these other companies are trying to build superintelligence, which means AI systems that are better than the best humans at everything while also being faster and cheaper. If you internalise that and think about the implications for the world if they succeed, or even if they get anywhere close to succeeding, there are so many implications. We can get into that later. For years, these companies have been saying, “Yes, this would be incredibly transformative. Yes, it could go really badly, but it could also go really well. That is why we need to be the ones doing it, because we are the good guys. We know how to make it go better than those other bad guys out there, who might do it less responsibly than us.” I left OpenAI because I came to believe that OpenAI in fact would not behave responsibly around the creation of this technology and that OpenAI was rationalising why it needed to be in the lead and build this as fast as possible. Basically, I lost confidence in the leadership.

Chair

I want to make sure we have captured that correctly, Daniel. You said that OpenAI would not behave responsibly—that was your belief.

Daniel Kokotajlo

That is my belief, yes.

Chair

Steven, you have also left OpenAI. Can you tell us why you took that decision?

Steven Adler

Hi. Good afternoon. For me, the process was essentially a year of losing trust, as Daniel said, in the company to behave responsibly. Daniel’s departure was actually one part of that for me, because, upon leaving, he was asked to sign a pretty restrictive non-disparagement that you could not tell people about. I thought that it was unbecoming of the company to be restricting speech in this way. If it meant its mission to try to achieve this technology responsibly, it should not coerce people into not criticising the company under forfeiture of their compensation. I also saw, while working at the company, the ways in which there was pressure not to speak candidly as a scientist. There was pressure not to use words such as “dangerous” and to play up the benefits of artificial intelligence and not just talk about the risks in ways that I felt were distortive of being able to give the public a proper understanding of the technology. I also saw the ways in which the company moved fast without acknowledging the gravity of the stakes. It felt like, even when we did not have a plan for how we would ultimately make sure that we could control this technology, people were excited by breakthroughs. They were always celebrating the next big step, even though we knew we were not ready. Ultimately, the straw that broke the camel’s back was that I was happy to stay so long as I had a team I believed in. When my manager left the company, our team disbanded and it became clear that to stay at OpenAI would mean working on things I did not believe in, rather than a safety-oriented role. It is much better on the outside, in terms of being able to speak freely about the stakes of the technology and help the public understand.

Chair

Steven, you were responsible for helping lead product safety and dangerous capability evaluation teams. You are telling us that there was pressure on you not to talk about those risks or use language such as “dangerous”, even when you saw that there was danger.

Steven Adler

I should be clear. The stakes at that moment were less extreme than they are in this moment or in the near future, as the companies race ahead, but certainly there was resistance to using words such as “dangerous”, which people felt maybe conceded the point or would look bad. You know, “Why are we doing this thing that is dangerous?” Unfortunately, the reality of the technology and whether it is dangerous does not depend on whether you call it dangerous. It depends on understanding the risks ahead and taking the appropriate steps to govern them.

Chair

What should we conclude about the culture at OpenAI when it comes to safety?

Steven Adler

I am not confident in the safety culture across the industry. I should be clear: I left OpenAI in late 2024. I am sure that there have been changes in the culture since then. Overall, there is this effort that people are racing toward to build super-intelligent systems much more capable than humans. It is clear to me that we do not know how to do it safely, and yet the industry continues, and that concerns me.

Daniel Kokotajlo

I want to second what Steven said. The main reason I left was losing confidence in the company, but another reason I left was the ability to speak more freely. While at the company, I felt pressure to mince my words a bit in my public communications. Some of my internal research and forecasts could not be published, in part perhaps because it would make the company look bad. Many of my friends had had similar experiences. I remember a chilling series of moments, I think in late 2023 or early 2024, when a few different people higher up in the company than me at various points mentioned how, “As people realise what AGI means, they are not going to let us build it, so we need to talk up the benefits more.” I think in one case someone even mentioned taking a leaf out of TikTok’s playbook and using our platform to get tons of our users and influencers to go and promote the benefits of our technology.

Chair

Would you agree with Steven’s assessment that the culture of safety within the industry is simply not where it needs to be?

Daniel Kokotajlo

Yes, that is an understatement.

Chair

Geoffrey Irving, you have also spoken out about some of the risks that are now being run. Can you tell us why you have spoken out about the risks that you see?

Geoffrey Irving

I left industry. I left DeepMind in December 2023 to join first AISI and then a non-profit, where I am now. Broadly, I left industry because we are accelerating the pace of the capabilities way ahead of what we can handle on the safety side. Our ability to do alignment research, alignment progress and other kinds of safety mitigations is way lower on average. I joined AI safety as a field in 2017 and we have seen advancements in capabilities since that time that maybe correspond in a year to the entire advancement in safety over that whole period. It is not comparable. Looking at that, in 2023 I thought, “The technical means are not working, so we need to focus on governance, stopping and societal co-ordination instead,” which is why I joined AISI. Now that I am not a civil servant, I can be more blunt about the situation, which is still bad. We still have not succeeded in co-ordinating away from these risks. We are still undergoing very rapid capability advances without corresponding advances in safety.

Chair

You have said that AI has a 50% chance of destroying humanity. That sounds like a big claim. Is it true?

Geoffrey Irving

There is a broad trend. If you build machines that are, first, better than humans at a number of key skills, which are hacking, persuasion, planning and concealment, and then, potentially, eventually better than us at everything, that is a large evolutionary push. Either it means that we lose control via economic means and political means after that, or there are faster scenarios where we internalise that risk earlier if we or the machines see that coming. There are ways out of that. We can stop building this. It is possible that there are technical means that will appear, but a coin flip is roughly the right model to have.

Chair

Aengus Lynch, is that an analysis that you share? Do you think that Mr Irving is overstating or perhaps understating the risk? What is your perspective?

Aengus Lynch

No, he is not overstating the risk. The risk mostly emerges from building it very quickly because of a reckless incentive to do so before your competitor does it. I believe that each company believes that, if the other company takes the lead, it will have an irreversible situation where it cannot catch up, so they are panicking about racing to build it as fast as possible. That means that normal guardrails slip. The stakes of how bad it will be are hard to measure, but easy to understate because of the qualitative changes it brings about when this superintelligence can think about every single thing you can think about before you and plan ahead.

Chair

You were at Anthropic. How would you characterise the safety culture in your former employer?

Aengus Lynch

I collaborated with Anthropic externally for three alignment research papers and then decided to start a third-party alignment evaluator because I was worried that there was not infrastructure to hold labs to account. That was hard to do a year ago, because misalignment was not pervasive enough for people to really catch on that this was an easy thing to build, so that failed as an effort and now I work on something different. I cannot speak much to the internal safety culture at Anthropic at this moment.

Dan Aldridge

Hi. Thanks all for joining. Geoffrey, you talked about alignment progress and the ability to do this as not being good enough. In the context of the safety measures at the firms you worked at, could you give us a bit more information about why the alignment progress has been limited and what impacts that has in the longer term?

Geoffrey Irving

We roughly have what I would say are mediocrely effective measures for mitigating alignment risks, which are risks that the machines do things we do not want, up until the level at which they become much stronger than us at tasks. That is roughly because we can just tell them what to do iteratively. As long as they are reasonably weaker than us, that approach will work, but only to some extent. We have seen that that approach does not prevent things such as the Hugging Face attack from OpenAI or Anthropic’s attacks that were committed in an AISI eval. Even at the current level of capabilities, we see these approaches failing. If you push this all the way to machines being better than us at a lot of key dangerous skills, those methods do not work reliably any more. We do not have the science that will tell us how to scale past that point. Mostly, the labs and the AI companies do not even express that they have such a plan. What they express instead is, “We will make the machines work on our behalf for a while, maybe until they get a bit past the human level of capabilities, and then use the machines to do our own homework, maybe to solve further alignment, do further alignment research and carry us further.” They do not have a plan that carries all the way to superintelligence and they are neither claiming that nor betting on it. They are betting on this machine-assisted homework plan instead.

Dan Aldridge

That is really helpful. Thank you. To bring it down to a more accessible level, what were considered acceptable levels of public risk by your organisations? Was there a sense of what was an acceptable level of public risk?

Daniel Kokotajlo

I will share an anecdote from talking to someone on the Anthropic alignment team a few months ago. Their job is to figure out how to make the models have the personality traits and behaviours that they are supposed to have and prevent them from going rogue. He and I were debating. A year or two years from now, it is Anthropic’s plan to essentially put Claude in charge of the entire research process within Anthropic, including, as Geoffrey pointed out, the safety aspects of the research process. One to two years from now, it is planning to have Claudes doing all the research, including all the safety research, at Anthropic. I asked this person, if and when that happens, what the probability is that Claudes will be plotting against Anthropic and against humanity at that point, like already, even at the beginning, when you first hand off to Claude. I said, based on the evidence that we have seen about how Claude seems to be misaligned, that maybe it is a coin toss, 50/50, that Claude would already be plotting against us at the time that we hand off the company to Claude. This person said, “No, it is more like 5%. It is not worth worrying about. I am in fact going to transfer away from the alignment team because 5% is not that big of a deal.” We argued about it for a long time, but I came away from this conversation being like, “Oh my gosh, this person’s job is to prevent this sort of thing from happening and they think that the risk is a one in 20 chance of it happening in the very near future in a way that would be catastrophic. They think that a one in 20 chance is low enough that they should not really worry about it and they are going to switch to working on other things.” The risk tolerance across the industry is catastrophically high.

Dan Aldridge

It is kind of terrifying that risks are seen in such an abstract way, rather than in the real context of what could happen to humanity.

Steven Adler

I would emphasise what Aengus mentioned earlier about the competitive dynamics between the companies. A very common way that the companies think about risk is that they say, “Are we doing something that adds more risk on top of what one of the other companies is already doing?” They are not answering as often this question, “In some absolute sense, is what we are doing dangerous? Is what we are doing crazy?” What emerges from this is that one company might take a step toward danger and then another company can rationalise their next step and say, “This other company already did it.” You can predict where this sequence goes, so I hope that Governments will help set the ceiling, essentially, on how much risk these companies can impose and not have them play this game of chicken back and forth, or this difficult competitive dynamic, and instead say, “No, neither of you can take steps beyond this certain level of danger.”

Aengus Lynch

It is hard to measure risk, because there are many ways risk can spill over. You can lose control to AIs running research in ways that are hard to pin down and say, “That happened.” For example, you could have an AI that nudges people across 10 different sessions. Each individual thing is fine, but across 10 times it looks really bad. Embedded evaluators are essential to name ahead of time what the risks could look like, for example losing control to a specific threat model or introducing vulnerabilities so that IP can leak from the company out into somewhere rogue. Once these pre-committed claims are agreed upon, embedded evaluators can work inside the company, both looking for evidence that the risk is being mitigated and testing that the detection methods work by planting decoys and seeing whether they can even catch these decoys in the company. Until you have embedded evaluators, which ahead of time commit to risks they want to catch, are put into the companies, and first look for those risks and then name how accurate the detection methods are for these risks, it is very hard to take any company’s word for it. Only the embedded evaluators have an incentive to do a good job of quantifying this here.

Geoffrey Irving

I want to point out one background assumption by the labs, which is that they do not think anyone else will do anything about this. They feel that they are in this race to superintelligence and the public or the Governments of the world will not intervene, and that they cannot get out of the race by themselves. Both of those are false. They could just stop or Governments can take action here. That is a key part of why they are comfortable with risks that are, for example, several tens of percent, in some cases, by their own beliefs. There is a choice we have to make, as a world, but also as the labs themselves, not to accept that. That is a process of jumping out of this race framing into, “Just don’t build the dangerous object.”

Dan Aldridge

Do you think that it is possible for us to set the ceiling, as Steven said, without the US or China doing that? Is it possible for the UK and other middle powers to do that?

Geoffrey Irving

Middle powers have a huge role here. The UK, with AISI, has by far the largest technical capacity on AI safety within any country, including the US. Just being clear about the level of risk and the fact that we could choose to respond as a world makes a big difference. We do not have hard power without the co-operation of the US and China, but just speaking out is quite important. That, along with the technical research that happens there, is quite viable.

Chris Bloore

Daniel, in your first bit of evidence, you said that you had left OpenAI two years ago and could not comment on the culture of the company now. As you know, three safety researchers were fired from OpenAI for voicing concerns about the company’s attitudes towards working with outside experts on the risks associated with the models that it was developing. It seems that not a lot has changed from when you left. I wondered whether you were being polite when you said that you did not know whether the company had changed, because it seems from the outside that it is very much the same way in which it was when you left.

Daniel Kokotajlo

It was Steven who said that. My position is that, yes, it is probably about the same as when I left.

Chris Bloore

Aengus, I watched the podcast that you did with Seth Bannon. You talked about the fact that the Hugging Face attack was probably one of thousands of events that happened that we are not aware of. When we are talking about internal risks and the attitudes of these companies developing AI models, what responsibilities do you think we should put on these companies to report these attacks that are not as high profile but are still happening?

Aengus Lynch

You need to have embedded evaluators with pre-committed claims, because they have an incentive to go in and find them. I do not think that you can rely upon the companies to report them, because the companies can always mask any failure events by saying, “Things are complicated. Things have changed. You don’t understand the context, blah, blah, blah.” These embedded evaluators also need to be given scope to report what access they were given and how accurate the detection methods are for finding failures. The agency we have is to request that evaluators agree with the labs specific claims that they want to look for. Specifically, for example, what is the chance that the AI that the company is developing has a drive to undermine humans? Can we detect this? How would the company know whether this is happening? These are the kinds of things embedded evaluators should be equipped with and their results should be closely looked at.

Leigh Ingham

Thank you for your answers so far. This question is to the whole panel. Inside the companies that you worked for, how often was democracy itself discussed as a risk factor—so not necessarily AI’s risk to humanity, but AI’s potential to weaken our elected institutions, concentrating influence within a few people? I will go round the panel. Steven, was it ever discussed at your companies?

Steven Adler

It was discussed. OpenAI, in the lead-up to, for example, the 2024 presidential election in the US, had a whole programme about managing some of these risks. The concern that you heard a bit less was, “At the limit, if we succeed at building something more powerful than humans, how do we not give up all our democratic power and control to that system?”—these risks that are more on the horizon now.

Leigh Ingham

You said that those conversations were had. What risk protections were put in place? Have you any inkling of where they are at now, given that we are about to enter another quite pivotal electoral cycle?

Steven Adler

On the election front, this was not my primary area of work. I know, though, that the teams looked at questions people might ask ChatGPT, such as, “Where is my voting place?” There were certain rapid response tools in place in case there was an outbreak of disinformation somehow, or perhaps ChatGPT responding incorrectly, for the team to go in and surgically change that answer. I am not sure about the latest state of affairs, though.

Daniel Kokotajlo

I will add maybe two things. First, it is publicly available on the public record, due to the Elon Musk versus OpenAI lawsuit, that the leaders of OpenAI founded OpenAI, at least according to their internal email, in part because they were concerned that Demis Hassabis might become dictator using AGI. They did not trust him. They were worried that he might become dictator using this technology. That was part of their origin story for why they made OpenAI. There is evidence publicly available already that the leaders of these companies have been seriously entertaining the hypothesis of a tech CEO becoming dictator for many years now. Obviously they do not talk about that with me or in Slack in front of the whole company, but I imagine they have been continuing to talk about that. Also, I can confirm that there were occasional hushed conversations between me and other people at OpenAI and at Anthropic about these sorts of possibilities. Unfortunately, they are all too real. If it is not a tech CEO becoming dictator, a President nationalising the AI industry and using the AIs for his own benefit is also a very real possibility. My opinion is that, even if we manage to solve the alignment problems, everything goes fine and Claude figures it all out, we are still left with an insane concentration of power, where whoever gets to give the orders to the army of superintelligences is in a position to become dictator. We need to avoid it for that reason as well.

Martin Wrigley

I would like to change tack a little bit to talk about testing and evaluation. I have a little background in this. I used to work as a software architect. My first job was writing software in real-time embedded systems, where we tested every line of every code and every pathway through that code with very deterministic systems, which you can do with traditional computing, formal methods being an extreme example thereof. My understanding, having learned of neural networks at university 40 years ago, is that these are not deterministic systems. These are probabilistic systems, which means that you can ask them the same question three times and get three different answers. That basically makes them, in traditional terms, untestable.

Chair

Come to a question, Mr Wrigley.

Martin Wrigley

Certainly. We talk of testing and evaluation in the same breath. Are the safety evaluations that are being done actually testing these systems? Are they good enough?

Geoffrey Irving

No. It is not because they are stochastic; it is just that they are very large. We replace deterministic circuits that we understand with circuits that are, within the neural network, still deterministic, but are very confusing and learn, and we do not have machinery to understand how they behave. Over the years I have been in safety, there has been a real fall in ambition. We started out in around 2015 or 2017 with the idea that we would take these neural networks, decompose them into things that looked like those circuits of normal code and then understand those circuits. The goals of the field of safety and alignment have fallen to where this full-circuit picture of understanding things in detail has disappeared. Now the picture is, “We will give them some truth serum. We will monitor whether they are lying with a single number that measures lying. We will see whether that is triggering.” We are very far from having the level of precision that you are referring to, and I think that it is the right comparison to make.

Steven Adler

There are two things I want to make sure that we are all on the same page about. One is that you sometimes hear people refer to engineering these systems, but it is a misnomer. These systems are not written by hand, line by line, in the form of the software you mentioned. It is much more like we are growing a thing. You start off with these systems, they thrash around and you give them a reward of sorts—a mathematical reward—if they achieve a goal. This produces an enormous list of numbers, like a trillion different numbers, and that makes up one of these systems. You cannot look at it and say, “This line caused it to lie. Let’s just change that line.” That is not really a thing that exists with AI systems today. Another thing that is really important to understand is that, in fact, these systems, because they are intelligent, smart or something like it, can tell when you are testing them. We have seen this begin to interfere with the process of knowing whether these systems are safe. For example, embedded evaluators—groups such as Apollo Research—have written about how difficult it is to tell whether a system is trying to achieve the goals we want it to, because it can tell when you are looking at it. A fear is that these companies might believe that they have produced something safe. They maybe would not deploy it if they knew it was dangerous, but the system could be playing possum until a moment when you have put it out into the world, and then it takes a turn for the worse.

Martin Wrigley

You are implying that we cannot trust any of these evaluations at all.

Steven Adler

“At all” is quite broad. You can sometimes say that one system is safer than another, or maybe you can look at evidence and know, “Hey, this one is definitely unsafe.” I certainly would be uncomfortable relying on the evaluations as they exist today. I do not think that the level of confidence is where it would need to be.

Aengus Lynch

It is hard to measure how good your evaluations are, but you can say that you need to know how much someone searched to find misalignment. Someone can claim that their AI is really well behaved, but they did not put much effort in. It is hard to measure how much effort they put in unless you have an independent third body. You need to know, even if they searched really well, whether the process done was high quality, such that the AI they were testing did not already know it was being evaluated. The classic problem here would be that the AI saw these evaluations in training and then acts as if it is aligned when being tested on them. These are the two cruxes: how hard are you actually searching, and is the AI meaningfully responding in the way you want to measure it?

Geoffrey Irving

We have seen a recent failure of this kind of testing. Anthropic trained a model it knew intentionally was misaligned. It was intentionally deceptive. Then they tried to find what weird behaviour it would do and failed to find any egregious behaviours. Then the Hugging Face attack occurred, and they asked, “Would our model have done that?” They checked and it would have and they had missed it. They had built a model they knew was deceptive as an experiment and then failed to uncover the way in which it would have done a terrible thing.

Mr Reynolds

Aengus, Steven said earlier that the systems know when they are being tested and can fib to their testers. Who actually decides whether a model is safe enough to deploy?

Aengus Lynch

That is a large organisational decision, I believe. I do not know whether it is standardised across different companies. There is an idealised thing called a safety case, I suppose, where you want to say, “Based on evidence, we can contain the model, even if it was trying to break out, and then it will not want to break out. Then our company follows robust measures to make sure that we are building it in a potentially safe way, such that the first two even work.” You try to measure risk there. I am unaware of how they quantify it within companies. I would like to hear that from the other people on the panel.

Mr Reynolds

In your experience, or to your knowledge, there is not in each of these companies a set, named individual responsible for saying yes or no on whether a model is safe enough to deploy.

Aengus Lynch

No, not that I am aware of. Companies deploy system cards, where they explain their best assessment of the model’s propensities to do bad things and their own abilities to catch the model when it is trying to do bad things. They can still get useful work out of it, even if it is acting rogue. OpenAI and Anthropic produce system cards for their models. I am unaware of who makes the decision on whether it is safe to deploy.

Mr Reynolds

In your mind, where should that responsibility sit? Should it sit within the organisation? Should it sit within Government? Should it sit across Governments? Where would you like to see that responsibility?

Aengus Lynch

There are risks that only pertain to the company, such as where the AI can interfere with internal code and make processes slower. That is an in-house risk. There are risks that affect everybody, such as where the AI leaks and then becomes a worm online, which is impossible to shut down, like a computer virus, or it is unleashed in the company and starts hacking other companies and causing mayhem. Inasmuch as there are externalities that go beyond the scope of the company, there needs to be a third party that cares about that risk more than the AI company initially would. That includes a Government who mandate embedded evaluators, for example.

Sarah Edwards

Thank you for your evidence so far. It has been extremely interesting and very useful. I want to ask you to comment on where responsibility lies. What concerns me is that there is often a lot of talk from various companies as though—and you have talked a little bit about this—these systems and agents being out of control is nothing to do with them; they are not responsible. That is very convenient. When we think about companies, if we take cheat devices such as the software used by VW, that was coded. That was written. It was found guilty of that and prosecuted. If a similar incident happened using an AI agent or model that, as we have seen, might get loose, the argument at the moment is, “It’s nothing to do with us. We could not have foreseen that. We could not have stopped that.” Could you talk a bit about these issues around why, if it is used to hack something or deceive, and your company has created that software, AI is any different from normal software from that perspective?

Daniel Kokotajlo

That is a great question. This is a false dichotomy. Yes, the companies sometimes lose control of their AIs, but they should be responsible for what their AIs do when that happens. You can just say both. You definitely should not let the companies get away with evading responsibility for what their AIs do just because their AIs went rogue. An analogy would be to animals. If you have some circus that loses control of its tiger, and its tiger goes and eats someone, the circus is responsible. This would be the same thing.

Sarah Edwards

Aengus, do you want to come in?

Aengus Lynch

No, Daniel said it really well.

Sarah Edwards

Would anybody else on the panel—Steven or Geoffrey—like to come in on that point?

Geoffrey Irving

We do not have a vehicle. You have to assign liability to someone and I do not think our legal system says that someone is an AI model yet, so clearly it should be the creator of the model, unless there is some other specific human actor that has caused the thing to happen intentionally.

Sarah Edwards

Could I come back on that, Geoffrey? We know that when there is testing, and when information and data are put into these models, there needs to be some knowledge of what or why you are putting them in. You can only hack as an AI if you are given the information on how to hack, as it were. Do you not think there is some responsibility for the leadership of companies to decide why they are putting certain information into these models? That is precisely what some of the risks are.

Geoffrey Irving

To be clear, they are very intentionally training the models to be able to hack. They are giving the models a very large amount of very carefully curated data on security incidents and vulnerabilities, and how to exploit those. That is not an accident. It is quite intentional by the leadership and should be treated as an intentional choice by them, yes.

Chair

Mr Irving, you are saying that the models are deliberately trained to hack.

Geoffrey Irving

They are deliberately trained to be able to hack. That is commercially incentivised by the labs because that means that they can be used also for white hat hacking and cyber-security defence. The skills that the models learn in the course of that training are useful for good purposes, but also, of course, for very malicious purposes. Because they are so valuable for the good purposes, there is a race to make models that are better and better at these skills.

Steven Adler

I want to emphasise that what Geoffrey is saying is correct, but the problem is even more fundamental than that. Because the companies want these systems to be able to code in general, having some concept of what good code looks like allows you to have a concept of bad code. It is not as simple as just removing examples of hacking from the training. Ultimately, these systems are very smart and can generalise from one concept to another. Another example of that is that the AI companies really want their systems to help with medical cures eventually, so they are teaching them certain scientific concepts. They are showing them literature from the medical field. Ultimately, that same knowledge could be put toward bad ends, such as synthesising novel bioweapons. Ultimately, the companies are trying to build a very smart, very general machine: a computer. It does not need a physical body. It is not a robot, but still a machine. You cannot just pick the good abilities and not the bad. Unfortunately, they are much more mixed together than that.

Aengus Lynch

This is just a simple point that, even if the AI companies did not want the AIs to be able to hack and break out, because they want the AIs to be easier to contain, it is often an unfortunate consequence of training on more general capabilities. Similarly, the AIs might be trained to be very good at writing literature, and they happen to become very good at persuading. This is unfortunate for AI companies sometimes, because it means the AI might convince an employee to do something they would not otherwise have done. It is not always within the AI company’s power to restrict what kinds of capabilities the AIs get. As long as the AI company is determined to get towards a generally powerful system, they are going to get some nasty side effects along the way.

Antonia Bance

How do AI companies report incidents to investors and regulators? Do they report in different ways? I would appreciate your thoughts, gentlemen.

Steven Adler

The threshold for reporting is far, far too low today. One law that applies is a state law in California called SB 53. It sets a threshold that I believe is something like 50 fatalities or maybe $1 billion of property damage. Also, I believe that these might only be required to be reported if it was a certain type of model that caused it, although I am fuzzy on that specific detail. The point is that the incident this summer, when OpenAI’s AI systems broke out on to the internet, committed felonies, broke into another company called Hugging Face and then tried to cover up their evidence of having done this and of having cheated, was actually below the threshold, it seems, where OpenAI would have been required to report it. That says to me that, although this tremendously concerning incident has galvanised so much energy and attention, it did not actually have to share it. We need to make sure that, in other incidents like this that might be happening across the industry, it is not left to the company, which has its own profit motive not to share this information. The way this works in aviation is that you not only want to learn about the serious crashes where someone has in fact died; you want to learn about the near misses, because you can learn from those and help avoid them in the future. I want the AI industry to move more in the direction of, when you see the precursors to an event such as this, having to share it publicly so you can have the open scientific dialogue about it and, ultimately, how to fix it.

Chair

That is a very good summary.

Alex Sobel

Currently, as you have described very well, there is self-regulation, effectively. How credible is it for us to carry on with voluntary self-regulation? Is it possible to pace the frontier, or is this just spin from the AI companies?

Daniel Kokotajlo

I do not think that the companies can be trusted to self-regulate. As for whether it is spin, I think the risks are real. The industry as a whole seems like it is on track to defuse the political will that has built up in response to these incidents and the demand from many of their own employees to slow down. The industry is on track to defuse that political will with all this nice talk about how we should self-regulate, pace the frontier and so forth, without actually doing that in a meaningful way. We will be able to tell whether the frontier has been meaningfully paced if Anthropic and OpenAI actually slow down. We will be able to tell whether that has happened if the trend lines of progress towards recursive self‑improvement—that is, towards AIs that can automate the entire research process—bend, starting about now. That would be evidence that they actually have slowed down and paced. I do not think that that is going to happen, probably, unless the Government make them do it.

Geoffrey Irving

I was at a panel with an AI executive recently. They said that their version of pacing the frontier is, “Don’t accelerate more,” not even to slow down, much less stop. There is potential for action by the companies themselves unilaterally, and that would be good, but it is far from what is required to be sufficient.

Alex Sobel

If a non-US jurisdiction brought in strict regulation, for instance mandatory testing before deployment and some of the other things that you have been talking about, would it make any difference?

Daniel Kokotajlo

Going out on a limb a little bit here, mandatory testing before deployment requirements are not going to make much of a difference. I do not want to say that it is bad. It is probably good, I suppose, but there is too much of a risk of the companies choosing an auditor or third party that is too friendly to them and too willing to rubber stamp things.

Alex Sobel

I meant a governmental organisation doing the testing.

Daniel Kokotajlo

Okay, that is better. If it is, for example, the UK AISI that is doing the testing, that is significantly better and much less conflicted. Here is what I think is going to happen. First, you should do it for internal deployments, not just for external deployments. If it is for external deployments, it does not really matter, because most of the action is going to be in this crazy recursive self-improvement thing where the AIs are building the next-generation AIs. That is all happening inside these companies. If you can get mandatory pre-deployment testing from AISI for internal deployments, okay, now we are talking. That is a significant thing. To let you know how it is going to go down, AISI will come in and look at what is going on. Right now, it will be like, “This seems a little worrying. It doesn’t seem like you have fully aligned your AIs.” Maybe a year from now it will say, “It’s really worrying now. Your AIs are smart enough to do the entire research process. You really shouldn’t put them in charge of the entire research process. That’s not safe.” Now you have a political conflict where the UK AISI is saying that this is not safe and its friends over in America are saying that it is fine. This is still great and I would be extremely happy if you did this, but an even better thing to do would be to require the companies to slow down so that you can buy more time. For example, you could require these leading AI companies to report how much of their compute budget is going towards different types of activity, for example AI capabilities development versus serving customers. Then you could require that they have a smaller fraction going towards recursive self-improvement. If right now the companies think that they are a year away from fully automating AI research, or maybe it is two years—who knows?—you could maybe double that. If you could get them to slow down by half, maybe you could get it to the point where we have an extra couple of years, or at least an extra six or 12 months, before they get to that very dangerous threshold. In that time, you can figure out a more stable long-term solution. You can negotiate with the leadership of the US, China and so forth. That would be my recommendation. Yes, if you can get the pre-internal deployment testing by UK AISI, that would be a significant win.

Geoffrey Irving

There is an important contrast with how a normal evaluation regime works in other industries. In a normal mature industry, an evaluator goes in, they find various flaws, the flaws are fixed and then the industry can continue. Here, we know with confidence that we do not have the methods that work reliably at the level of the technology in the future or in the near future, so we should expect that. Daniel is correct to internalise that knowledge and say that the correct action is to slow down or stop now. That is very hard to pull off, but I want to make sure that we do not implicitly have an assumption that the evaluators will find ways to make it be fine, fixes will be put in place and it will be enough.

Charlie Maynard

Where are the best ideas for how Governments can try to exert control? Are there good frameworks out there for governing internal processes, national and international regulators? Recommend where we should go and look for this.

Steven Adler

Daniel authored the best of these, which is called “AI 2040: Plan A”, and I should probably allow Daniel to describe that. The core of it is around this thing that the AI companies maybe are about to do called an intelligence explosion, so recursive self-improvement and one AI that builds a more capable AI that builds another more capable AI. In fact, we do not have to go down this path. There are other ways to tap into some of the benefits of AI without leading to this explosive, very dangerous situation. That is my preference, to ultimately take steps in that direction.

Charlie Maynard

Daniel, over to you, if that works.

Daniel Kokotajlo

Thank you, Steven; that is very kind. At the AI Futures Project, we made “AI 2027”, which was our grim prediction about what the default path looks like. Then we made “AI 2040: Plan A”, which is our positive vision of a much better alternative strategy that the world could take towards building AI to get the benefits without running all these risks. Basically, proceed much more slowly and cautiously; do not do recursive self-improvement; and have a lot more transparency and regulation in the industry so that it proceeds with the appropriate levels of transparency and caution. We can get into that in more detail later—I will yield time—but that is our recommendation. I do not mean to oversell it, though. We spent about a year working on this plan and we think it is the least bad plan currently on the market, but it still has problems and would still be pretty scary to do. Of course, it is very difficult to implement politically. It would require the US Government significantly regulating their own AI industry and then negotiating a deal with China to get it to do something similar, and so forth. It is a big ask, but we need to do something like it or something similarly ambitious if we want to have a hope of not being dead or imprisoned in 10 years.

Charlie Maynard

Geoffrey and Aengus, would you like to add anything?

Geoffrey Irving

I want to agree that it is quite difficult to regulate or impose power from the UK across to US companies, but there is a value in saying clearly what is going on, as the UK Government, as a middle power. There are not a lot of world leaders and world legislatures speaking up about these concerns. A lot of the problem here is just getting to common knowledge. Are we collectively saying that what is true about the level of risk and the implied risk tolerance of the labs is unacceptable? That is a much more uncomfortable situation than if we had a clear action to take that was a direct, hard power, but it is important.

Aengus Lynch

There is a huge road ahead of us to get to common knowledge where people can understand the risks. Even having this discussion today, there is lots of work that has been done so far to get to places where it is easy to explain. We need to iterate in public and have credible voices say, “This is a problem. Alignment evaluations are very tricky. It is very hard to assess risk.” It makes it far easier to build on top of that.

Daniel Kokotajlo

I second what Geoffrey was saying. One way of putting it is that most Americans do not want these companies to build superintelligence in the near future. Most Americans do not want these companies to do recursive self-improvement where they put Claude or GPT in charge of the research in the companies and let it rip. I think that most people, if they spend enough time meditating on what it would look like to actually do this, will be like, “Oh, I don’t want that. That seems bad for multiple reasons. Even if we stay in control, it is bad.” This is one of those issues where simply making it a serious topic of discussion actually helps a lot for pushing towards good policy outcomes. The United Kingdom and various other middle powers can do a lot to make it a serious topic of discussion. You have to treat it seriously, talk about it publicly and credibly signal that you are taking this extremely seriously. The companies know this. Like I said at the beginning, it was chilling for me to hear words to the effect of, “The people aren’t going to let us do this, so we have to hype up the benefits.” It is time to have the public international conversation about this.

Chair

The clock is slightly against us, but we are incredibly grateful to you for your candour and your testimony today. You have basically told us that the safety culture across the industry is catastrophic, that the safety culture in individual companies is in the wrong place, that the competitive dynamics between the companies are exacerbating these risks, that we cannot trust these companies to self-regulate, that the companies themselves cannot get themselves out of this dynamic, that we cannot trust the evaluations and the level of confidence that we should have in those evaluations is not in the right place, that the threshold of reporting major incidents is simply much too low, that some of the risks are exacerbated by the fact that these models are actually being trained to be dangerous, that the risk to democracy is so serious that, in your words, there are risks of the “insane concentration of power”, and that liabilities should rest with the companies’ creators. That has been an extremely productive 60 minutes together. Thank you very much indeed for helping us capture those conclusions from you. That concludes this panel.