CFR Fellows’ Book Launch Series: The Infinity Machine

In The Infinity Machine: Demis Hassabis, DeepMind, and the Quest for Superintelligence, Sebastian Mallaby examines the rise of artificial intelligence (AI) through the story of Demis Hassabis, a pioneer whose work has helped place AI at the center of scientific and geoeconomic competition. Through extensive interviews with Hassabis as well as conversations with his collaborators, critics, and rivals, Mallaby traces Hassabis’s path from a chess prodigy in North London to a leader in the race to develop artificial general intelligence. Written for anyone concerned with the forces reshaping technology, industry, and global power, this book explores the promises of advanced AI as well as the growing debates about its long-term consequences. 

The CFR Fellows’ Book Launch series highlights new books by CFR fellows.

RUBENSTEIN: (Off mic)—David Rubenstein. I have the honor of serving as chairman of the board of the Council on Foreign Relations.

And today we’re going to talk about a new book by Sebastian Mallaby. The book is right here. I just finished reading it a couple days ago. I highly recommend it. Called The Infinity Machine: Demis—is that how you pronounce it?—Demis Hassabis, DeepMind, and the Quest for Superintelligence. And Sebastian, as all of you probably know, is the Paul Volcker fellow for global economics at the Council. He is a graduate of Eaton and first-class honors from Oxford as well. Eight years at the Washington Post as a columnist and an editorial board member, thirteen years at the Economist. And this is his sixth book. Previous books have been on Jim Wolfensohn, Alan Greenspan, among other things. And, most recently before this, The Power Law, which is about venture capital world.

MALLABY: David, you got all of that right, except you didn’t pronounce the second name correctly.

RUBENSTEIN: I’ve never pronounce it correctly.

MALLABY: You said “Hassabis.”

RUBENSTEIN: How do you pronounce it?

MALLABY: Hassabis.

RUBENSTEIN: Hassabis.

MALLABY: You have to put the emphasis on the right—on the right syllable. (Laughter.)

RUBENSTEIN: Hassabis. Just like—all right. I’m going to pronounce it correct eventually.

MALLABY: OK.

RUBENSTEIN: OK. So for those people that haven’t yet read the book, but I’m sure they’re all going to want to read it after this conversation, what is it about? The essence of it, just summarize it. Who is Dennis Hassabis? (Laughter.)

MALLABY: So Dennis Hassabis—(laughter)—is a classic product of London. He has a Chinese Singaporean mother. He has a Greek Cypriot father. He grew up with not much money. And he went to the local government school. And he was a genius. At four years old, he looked at a chessboard. He understood chess. By about twelve or thirteen he was the second-best player in the world in his age group. And chess kind of catapulted him out of this rather modest background. And he went from chess to then writing computer code to kind of—he was interested in chess programming. And then he wrote the software for a computer game when he was around sixteen called Theme Park, and it sold more than five million copies. Made him quite a lot of money.

And his boss, who was running the video game lab, said to him: You’re so good at this I want you to stay. Forget going to Cambridge University. Forget doing computer science. You know, how about I give you this check instead? And the check was for more than a million dollars in today’s money. Remember, he comes from a modest background. And he doesn’t cash the check. He says, I prefer computer science to your money. And he goes off. And he’s determined to build AI. Goes to Cambridge.

RUBENSTEIN: Did he go to Cambridge?

MALLABY: Yeah.

RUBENSTEIN: Did he do OK there?

MALLABY: Yeah, I think he did all right. You know, he did have a Porsche car, which—from his computer science successes. And so that attracted some attention. But other than that, he was a good student.

RUBENSTEIN: But as a chess player, he was basically the best junior chess player in England, is that right?

MALLABY: Yeah, yeah. He was captain.

RUBENSTEIN: And he played on the chess team for Cambridge? Every time they beat Oxford, he was the captain?

MALLABY: I think at the time he’d chilled a bit on chess and was interested in learning lots of different things. And one of the key things—I actually read recently a book called Range, about how people who do super well in life often have a big range of stuff they’ve tried. And Demis was a bit like that. You know, he went through phases. Chess was the first phase, but then there was programming. There was also theoretical physics. He became obsessed with biology. By the time he got to Cambridge, he was playing lots of other games, like go. He taught himself. And then he became a competitive poker player. And so he just took on all these different things.

RUBENSTEIN: OK. So let me ask you when did you first hear of him? Or what attracted you to write a book about him? And did he say, hey, I want to have a book written about me, and I’ll give you all the time you want?

MALLABY: (Laughs.) I’d seen him at tech conferences in Europe in the kind of 2010s. And he was always this striking person because he seemed boyish and approachable and unassuming, and then he would get on the stage and suddenly go from sort of Immanuel Kant to neuroscience to computer science. And these ideas would just gush out, from this kind of almost, yeah, boyish-looking figure. He was probably in his late thirties then, but he looked a lot younger. So I was always intrigued by him. And then I realized that, as AI started to really do stuff—particularly with the protein folding system in 2020 where all of the proteins in nature were unraveled—I realized that you could tell the whole story of modern AI through this person’s career, because he’d been obsessed with it when he was in his teens. And at that point, AI could do nothing. It couldn’t recognize a picture of a cat. And then through the period when he became more and more obsessed, it became stronger and stronger. And then he built all these systems. So that was—

RUBENSTEIN: So he was a chess genius, I guess. And then he was very good at developing computer games, or computer games of various types. And then he decided artificial intelligence is what he really wanted to focus on. And did he decide to work for somebody else, or did he start his own company?

MALLABY: Well, that was the interesting thing. He worked for somebody else for one year after college. And then he quit and started his own company. Now, in 1997 when he left Cambridge there was no Silicon Valley in Britain, entrepreneurship wasn’t a thing. He did it anyway. He’s sort of sui generis in many of these. I mean, being obsessed with AI before AI worked, like fifteen years before it worked, and then also being ready to start a company in England. And this was so counter-cultural that when he went off to raise money he found out there were some people who described themselves as venture capitalists in London. So he went to see these people. And one of them, you know, took him out to lunch, gave him way too much alcohol. And at the end of this, said, well, you seem like a bright person because you play chess. So I’ll offer you a job in my bank as a currency trader. And Demis like, no, no, I’m trying to start a company. And the guy wasn’t—you know, I mean, it’s just—there was no mentality of startups. Venture capital didn’t exist.

RUBENSTEIN: Some people, like Mark Zuckerberg, picked up from East Coast and moved to the West Coast. Did he think about moving to Silicon Valley?

MALLABY: You know, at the time—you know, this was the mid-’90s, late—mid-, late-’90s, Silicon Valley, America, that wasn’t on his map. I think he went to America the first time ever when he did a postdoc when he was quite a bit older.

RUBENSTEIN: Right. So he ultimately starts his company. What’s it called?

MALLABY: The first one is called Elixir.

RUBENSTEIN: All right. And how did it do?

MALLABY: So the idea was to build cool computer games and to incorporate early AI into the computer games. And it did OK, but not great. And the reason was that Demis was so ambitious about the level of the technology he wanted to put into these games that he killed the company by demanding too much from the engineering team. And he would—you know, he was—he was a genius at salesmanship. So they called him the Jedi, all his colleagues. And he would look in your eyes and say, you will believe these things. And then people would believe them. So even when the—even when the demo of the game at the biggest game festival crashed, it didn’t matter. Because the thing had crashed, but he told such a good story that everybody believed the story, and then they funded the game.

RUBENSTEIN: Right. So eventually, though, he starts DeepMind.

MALLABY: Right.

RUBENSTEIN: And where does the capitalization for that come from?

MALLABY: So this is now 2010. And by this time he has discovered that America exists. And he’s heard rumors that there are real venture capitalists in California. So he goes to San Francisco to the Singularity Conference, which is full of kind of AI aficionados before AI is working, and it’s kind of on the borderline between visionary and weirdo. And he meets Peter Thiel there. Rather fitting, maybe. And he pitches Thiel. He says, you know, the thing about chess, Peter, is that it’s all about the balance between the bishop and the knight. So very cleverly, he didn’t talk about his company. He hooked Thiel on a chess story. And then—

RUBENSTEIN: Peter Thiel was a chess champion as well.

MALLABY: Of course. And that’s—and then the next day—or then Thiel said, well, you’re interesting. Come to my house tomorrow. Then he did the real pitch. And he took two million bucks out of Peter Thiel, which for 50 percent of the company was a derisively small amount. But there was no plan B. There was no other person crazy enough to invest in an AI startup, when the first AI product was clearly more than a decade away.

RUBENSTEIN: So eventually the partners of Peter Thiel say, this is not a great idea. And eventually they don’t keep investing. Is that right?

MALLABY: Correct, yeah. I mean, where’s the product? And Demis would say, this is the biggest invention in the history of humanity. You want me to say what the product is?

RUBENSTEIN: So where does he get money to keep the company going?

MALLABY: So he has a bidding war going on. Google is very interested, because Larry Page’s father had worked on early AI as an academic. And so Google is keen. And then, you know, just to make sure that Google doesn’t pull out, and to kind of put the pressure on them to move the deal quicker, Demis has the idea that he’ll go and see Mark Zuckerberg. He doesn’t really like Mark Zuckerberg, but he goes to see him anyway just to get a plan B. And he plays this trick on him. He says—you know, he walks in, and this is a dinner at Zuck’s house in California. And Zuck says, AI, your thing. It’s amazing. I want to buy your company. Don’t sell to those other guys. AI at Facebook is going to be—like, we’re going to make it central. So Demis is like, yeah, yeah, yeah. And then after an hour or so, he raises a few other technologies. He says, yeah, augmented reality is cool. And Zuck goes, oh, it’s amazing. And then he says, yeah, 3-D printing is cool. Oh, incredible.

And so then Demis says, this guy’s just bullshitting. He’s not—you know, he thinks everything’s amazing. And so he puts that on the back burner, proceeds with Google, and then Elon Musk tries to buy him. Because Elon has this sort of will to win. He wants to be the dominant industrialist of the twenty-first century. In case any of you hadn’t noticed, that’s his agenda. (Laughter.) And so he hears—you know, he meets Demis because they’re both funded by Peter Thiel. And they had this interesting discussion where Elon says, well, I’ve got the most important project in science going on in the world because, you know, if we become a multiplanetary species with Mars, you know, then we’re fine because if anything goes wrong with Earth we’ll just go to Mars. And Demis says, yeah, well, I’m doing AI. And if my AI wants to follow you to Mars, it’ll just build its own rocket and follow you. Elon goes, oh. And there’s a long silence. And he says, I think I want to do AI, too. So he tries to buy Demis’ company.

RUBENSTEIN: And that doesn’t work.

MALLABY: No, because Demis points out to him that Musk doesn’t, at the time, have any serious compute power. And if you’re building AI, you do need a lot of computers.

RUBENSTEIN: So eventually he goes back to Google.

MALLABY: He goes back to Google. He sells to Google. He sells to Google for 400 million pounds, $658 million. And the conventional sort of line amongst tech people in London is, oh, we sold to the Americans again too early. We could have had a national champion. I say, no, rubbish. You persuaded the Americans cunningly to pour nearly a billion dollars a year into your R&D in London. You built up your R&D system in London with these guys in California financing you. Victory. But, anyway, that’s what happened.

RUBENSTEIN: And what year did that happen, the investment?

MALLABY: OK, so that was the beginning of 2014.

RUBENSTEIN: 2014. So the company’s then four years old.

MALLABY: Correct, yeah.

RUBENSTEIN: And then the company continues to grow. And talk about two of their products. One is designed to beat chess players. And is that that hard to do?

MALLABY: (Laughs.) Actually, so what happened was that they—chess had been done before, 1997 Deep Blue. So they did go. They moved on to go.

RUBENSTEIN: Well, they did one for chess too, first.

MALLABY: Yeah. They did Atari, actually. It was the computer games suite called Atari.

RUBENSTEIN: For those who aren’t familiar with the game go, much more complicated than chess, correct?

MALLABY: Right, right. Because, you know, chess is only an eight by eight board. I mean, that’s kid’s stuff. You know, go is nineteen by nineteen. So when the first person does the first move, there’s 361 different places you can put that circle down. And then the second is 360, the options left. So you do 361 times 360 times 359, you do a factorial, and pretty soon you have this almost infinitely big number. And so to make sense of that decision space is super difficult.

RUBENSTEIN: All right, so to see that they could do something with go, they have the leading go human in the world who play against their artificial intelligence machine. And what happens?

MALLABY: So they do this in Korea. They cleverly stage it with all the media they can find. They put a $1 million prize on the table, just to lift the excitement. And they really tried to turn this into the new equivalent—because the 1997 match between Garry Kasparov and the IBM System Deep Blue had been quite a media sensation. So they wanted to 10X that, and they pretty much did. I mean, it became, you know, that was when AI went from total nerd obscurity to being somewhat notorious.

RUBENSTEIN: So in the famous IBM match, he was against the leading chess player in the world at the time. And everybody was surprised that IBM beat the greatest chess player. So beating the greatest chess player with new artificial intelligence isn’t as significant as somebody beating—as the computer beating go, because go is much more complicated.

MALLABY: Correct. Go is more complicated.

RUBENSTEIN: All right. And after they do that, they then take on another challenge. And that’s proteins.

MALLABY: Yeah. I mean, there were a couple of things on the way, but, yes, basically the next big one was a system that took the idea with go, which is this vast combinatorial space, and they chose an even bigger space. And this time it was all the different shapes into which an amino acid strand could fold itself. And, you know, basically you think of this strand, it’s going to do a self-executing origami trick, making this fantastically intricate, beautiful shape. And these—and the resulting shapes are proteins. And that is the building blocks of nature. And if you know the shapes of all the proteins, you can then invent medicines that will bind onto the protein and be pharmaceuticals.

RUBENSTEIN: All right. So they take on that challenge, which many people thought was beyond human capabilities. And what is the result?

MALLABY: So after two years they got to the stage where they had the best system in the world for predicting protein shapes. And because there were other computational teams in academia and so forth. And they beat all of them. And then there was a very interesting argument, because within the DeepMind team the boss of the protein folding team, the chief scientist, said: Look, Demis, you know, we’re the best in the world. Let’s declare victory and move on. And Demis is like, no, that’s not the point here. The point is we need to do something where we really discover all of the shapes with a lot of accuracy, so that now we can then build medicines on top. And the head of the team said, impossible. And Demis has this test of figuring out whether he should invest more resources and insist on continuing, or whether to quit. And remember, I said earlier, with Elixir, his startup, he pushed the team too hard and then actually killed the company.

This time he had this trick. He sat in the meetings with the scientists and he listened to them debate new things they could try scientifically to make the system get better. And he listened out for what he called the fluency, or the fluidity of the discussion. If the discussion was very fluid, it meant there were lots of possible avenues of exploration. And so because the fluidity was high, he said, right, we’re going to go forward. He switched out the leader of the team. The guy who said it was impossible left. A new person ran it. And then that person cracked it. And by 2020, so four years after they began, they had cracked this thing. And they won the Nobel Prize.

RUBENSTEIN: Won the Nobel Prize in chemistry. It’s pretty significant to be building artificial intelligence company, on the side you win the Nobel Prize in chemistry.

MALLABY: Yeah, and you said to me earlier on, so why pick Demis as the central character? I mean, Sam Altman does not have a degree, let alone a Nobel Prize.

RUBENSTEIN: OK, well, let’s speak about Altman and OpenAI. So you have DeepMind. And they’re building up their artificial intelligence company. When did OpenAI start, and who helped start that?

MALLABY: Right. So, remember, DeepMind started in 2010. After five years, stuff is working. And as I mentioned, Elon Musk wants to get into the game. In 2014 he’s been stiffed. He’s been told he can’t buy DeepMind. So now, in 2015, he needs a new plan. And this silver-tongued friend of his, Sam Altman, puts the idea in his ear. Well, we could do our own AI rival. We could compete with this Google thing. Let’s do—we’ll be—we’ll make it more virtuous, because Google is an evil, profit-making company. That was their line. And so evil, profit-making company is very bad. So AI should be for humanity. It should be open and shared with everybody. We’ll call it OpenAI. And so there was this coming together of Sam Altman and Elon Musk to set up the first rival.

RUBENSTEIN: But I thought what they were also focused on is they were afraid that artificial intelligence, if controlled by Google or any other for-profit company, could ultimately create something that would destroy humanity. And these guys, OpenAI, didn’t want to do anything that would hurt humanity. Is that right or not?

MALLABY: That’s true. It’s also true that every single one of the early AI labs began with people saying, we mustn’t destroy humanity. I mean, I thought before I began the project that, you know, it’s interesting the parallel between AI and nuclear weapons and Robert Oppenheimer, but I should be a little bit tactful about how I brought this up. It turns out you don’t need to be tactful because they bring it up for you. They—you know, Sam Altman says, you know, what my birthday is the same as Robert Oppenheimer’s. Did you know that? I mean, he actually volunteers it.

And Demis is the same. You know, you ask him, well, what was it like to have your first DeepMind office in London? He said, was great because, you know, you came out and then just on the corner there was this pedestrian crossing, black, white, black, white. And you know what happened there, Sebastian? Well, a Hungarian nuclear scientist, Szilard, was crossing the pedestrian crossing, had the idea for the nuclear chain reaction right there outside my office. And that’s what led to the Manhattan Project. So it was perfect, because we are the new Manhattan Project.

RUBENSTEIN: But for people who aren’t that familiar with AI, what is the theory by which AI could eventually eat up the entire world. Destroy it by doing what? How would that happen?

MALLABY: There are many forms of consumption, but I’ll give you the two main ones, right, for eating up the world. I mean, there’s sort of the bad guys get hold of the system and are empowered by it. And they either do some kind of massive cyberattack. And together with colleagues from CFR, I actually participated in a scenario where that was what happened. But there’s a massive cyberattack. It’s frozen all the infrastructure. It’s a total disaster. None of the lights are working. People are getting hungry. And this is because some unidentified bad actor has gotten hold of the technology. So that’s sort of category number one.

Number two is the machine itself turns against you. That’s kind of the Terminator thing. It sounds like sci-fi. And I went around for the first couple of years of this project thinking to myself, it’s fine. These systems will be smarter than us, but they don’t have an incentive to mess us about.

RUBENSTEIN: All right. So OpenAI is getting started now to do a not-for-profit kind of discovery of OpenAI—of AI. How is it proceeding? And how is DeepMind competing? Is it worried about competition?

MALLABY: This is now, you know, end of 2015, going into 2016. And DeepMind is super chill, right? I mean, they’re a little worried that this guy, Ilya Sutskever, who’s a genius, has agreed to leave Google and join OpenAI as a chief scientist. And they tried hard to stop that. But in the emails that have been leaked as part of the discovery and now the lawsuit between Elon Musk and Sam Altman, you read them all saying to each other, well, you know, Ilya agrees that Google is not virtuous. We are the small startup. We’re the virtuous one. So Ilya is going to come and work with us. So DeepMind was a little perturbed by this, but basically they were way ahead and they weren’t too worried. And they carried on not being worried really until ChatGPT.

RUBENSTEIN: Yeah, when ChatGPT came out it was the third version that caught all the attention.

MALLABY: Yeah, so three and a half. Yeah.

RUBENSTEIN: OK. So it came out and the world said, what is all this? And everybody said, who is DeepMind? We don’t know about them, but we know about OpenAI. What was DeepMind’s reaction? What was Google’s reaction? They’ve been working on this a lot longer.

MALLABY: I went to see Demis Hassabis right after the ChatGPT announcement, when it was really going viral. And I said, well, what do you feel? And he said, this is war. He’s a pretty competitive guy. He said, they have parked the tanks on our front lawn, which being translated into American means the front yard.

RUBENSTEIN: OK. So what did they do about it? Did he say, I got to work harder?

MALLABY: So, well, then comes the—you know, what I think will be a business school case study, because Google at the time had two different AI labs. It had Deep Mind in London, which was off doing the most sort of theoretical, cutting-edge stuff. And then it had some sort of academic projects at Google Brain in Mountain View. And Sundar Pichai, who people underestimated massively, who seemed kind of a bit quiet and softspoken and not forceful enough, immediately said, right, we’re merging these two entities.

Now any business school conventional wisdom would have said, look, mergers are difficult. You’ve done a few mergers, maybe, at Caryle so you know about this. Mergers are difficult. You’ve got different cultures. In this case, you’ve got two teams that have competed against each other. They don’t like each other. They are eight time zones separated from one another. You think you’re going to overtake OpenAI by putting these guys together, squishing them together? It’s going to be chaos. It was not chaos. In two and a half years, they overtook OpenAI. So this really was an amazing catch up. And I think the key to it was going back to that game design background that Demis had.

RUBENSTEIN: All right. Today what is the status of DeepMind? It’s now part of Google, still? And they did try to buy it out, but that didn’t work, right?

MALLABY: Yeah. So it is now part of Google. It’s called Google DeepMind. And it’s kind of the intellectual engine room building all of the AI. There was a phase, and this upset Demis a bit. You know, I did write a whole chapter about his lengthy attempts to get his way out of Google, to split out, to spin out. He didn’t like it when I wrote about that.

RUBENSTEIN: All right. So today how did you get access to all the people? Did you just say, look, I’ve written five other books. I’m a pretty well-known person. I’m part of the Council on Foreign Relations. And did that get you in?

MALLABY: They said to me, you know, yeah, Council on Foreign Relations, that’s Rubenstein’s outfit, right? (Laughter.) So then it was fine.

RUBENSTEIN: So you got in?

MALLABY: Yeah.

RUBENSTEIN: And how many hours did you have with Demis Hassabis?

MALLABY: I started—I lost count after thirty hours. But thirty and a bit—I don’t know, thirty-five.

RUBENSTEIN: And but you didn’t talk to Altman, right?

MALLABY: I did not talk to him. I did speak to a hundred people in the entourage, kind of both at DeepMind and outside DeepMind, including some people who worked with Altman, but didn’t talk to him.

RUBENSTEIN: So how many years did it take you to work on this book?

MALLABY: Nearly four.

RUBENSTEIN: And it took you two years of research, or three years of research, and one year to write, or something like that?

MALLABY: Yeah, although it’s not the case that you can neatly sequence these things. You have to be writing while you’re researching.

RUBENSTEIN: So who was the smartest single person you met in the course of this book?

MALLABY: I think—well, if defined smart in a quite a broad way I’d say it’s Demis. I mean, he really combines that Nobel-quality science, with an ability to lead a company, with an ability to relate to everybody. And he’s extremely approachable and easy to talk to. So I think the kind of full package he’s exceptional. But, you know, it’s hard to tell. If you just say, as a scientist, there’s a man called David Silver who was the key player in a lot of what DeepMind did. The person who also won the Nobel Prize for Chemistry with Demis for the protein folding is called John Jumper. He’s amazing. Ilya Sutskever is very impressive. There’s lots of others.

RUBENSTEIN: So who do you think made the biggest mistake by quitting a company and trying to start their own company, didn’t get anywhere? Or who left the most money on the table?

MALLABY: (Laughs.) That’s a great question. So, as I said before, I don’t agree with the analysis that said Demis did that because, you know, he shouldn’t have sold. You know, there are people who try to argue that he could have gone and stayed independent and built his own version of Google. But I think he needed so much capital to do the research that’s not really correct. So I think there’s a wave of people who have left the labs in the last two, three years who are great scientists, that they were able to raise money and they won’t necessarily build great companies. Maybe Ilya Sutskever is one. An exceptional scientist, not necessarily an obvious company builder.

RUBENSTEIN: So why do you think Silicon Valley has taken off and is the place where people go to raise money and all these companies have done very well, and England, the company of your birth—country of your birth—has not done as well in the venture capital and high-tech world?

MALLABY: Well, this goes back to my previous book about Silicon Valley and venture capital. And my view really is that, you know, what makes Silicon Valley stand out is the network. It’s the ability of ideas, people, and money to circulate and combine in lots of different iterative experiments, which you call startups. And the reason you have that ecosystem in Silicon Valley is because of the venture capital, because the VCs are the people who are incentivized to be the bees who, you know, move around between the flowers.

RUBENSTEIN: All right. Of all the people you met for this book, if somebody came to you for money for something they wanted to start as a new company, who would you give money to?

MALLABY: Wow. I would say—so, I think David Silver, who has actually just done this, was a good bet. He’s now being backed by Sequoia. Maybe he came to you for money. (Laughter.)

RUBENSTEIN: And one of the people in your book—

MALLABY: If you backed him, it was a good move.

RUBENSTEIN: One of the people in your book was there at the beginning. Now he left DeepMind. He’s now at Microsoft, right? Is he pretty talented as well?

MALLABY: Yeah. So very different character. Mustafa Suleyman. Another north London product of the—you know, the British melting pot. Father was an illiterate Syrian cab driver who had no—there were no books at home, no music at home. They went to the mosque every Friday. Mother was a convert to Islam. And they split up, his parents, when he was about fifteen. And the mum went with her new partner to go live in New Zealand. And the father said, well, I need to find a new woman and I can’t find one in England because I don’t speak good English, I’m going back to Syria. And left Mustafa and his fourteen-year-old brother to fend for themselves in Britain.

And so Mustafa had the character and the smarts to survive basically just being abandoned. Got into Oxford. But hated Oxford so much because it was full of kind of fancy people and he felt, you know, different, that he quit Oxford and set up the Muslim Youth Helpline and tried to prevent distressed Muslim youth—in a period of Islamophobia right after the Iraq War—he tried to prevent young Muslims from committing suicide. That was his mission. And so pretty amazing to go from that to being a cofounder of DeepMind, and then to be the CEO of AI at Microsoft.

RUBENSTEIN: So if you were the father of a young child today, or the mother of a young child today, would you say: Play chess, dear little child, and you’ll grow up to be really famous and rich? Or you think chess can drive you crazy and you wouldn’t want them to do that?

MALLABY: That’s a good question. I do think it was probably on the crazy end of the spectrum. I mean, the stories of the wooden boards under the tables when the kids played against each other, and the boards are there because otherwise you kick the other guy when he’s about to do a move, and you try and just—you know, I mean, it was vicious. And so I’m not sure—you know, I’m too soft in the twenty-first century to believe that would be a good thing.

RUBENSTEIN: So after doing this research, when you have a question do you go to ChatGPT or do you go to DeepMind? Where do—what do you use?

MALLABY: I do use Gemini.

RUBENSTEIN: Gemini.

MALLABY: Yeah.

RUBENSTEIN: You think it’s better than ChatGPT?

MALLABY: My honest opinion is it’s probably similar. Like I think Claude is also very good. You know, it depends on the use case. For coding, apparently code is the best. I suspect that whichever one turns out—if you did all—looked at all the clever benchmarks, and you said, OK, today Claude is the best, it would be different in three months’ time. If you’d asked the question back in November of last year, Gemini was thought to be the best. Now it’s probably Claude. They’re super close.

RUBENSTEIN: OK. All right, I missed the artificial intelligence development. I missed it, OK. But what’s the next big thing that I can invest in? (Laughter.) What can I—what should I—what’s next? What are you going to do your next book on that I can start investing in that now? What is that?

MALLABY: I think one day there’ll be a book on quantum. It may be a little early. Maybe I need to go do one about private equity first, and then I’ll come back to quantum. (Laughter.)

RUBENSTEIN: If you find any geniuses in private equity, let me know. (Laughter.) OK, we have time for questions. So we have thirty minutes allocated. And we’ll have some from online and some here. Please raise your hand, stand up, give your name and identification, and then ask your question. Thank you. Right here. Start here. Right here. Just stand up, give your name and your identification. Take the mic.

Q: Joe Gasparro, Royal Bank of Canada. Congrats again on this incredible book.

So you’ve now written the definitive histories of hedge funds, venture capital, and now AI. As you look at the titans in each of these arenas, how are they different? Maybe it’s the way they assess risk. But also, how are they similar? Are there certain reoccurring archetypes or traits that they all share? Thank you.

MALLABY: OK. So, you know, venture capitalists are the odd one out. If you say AI, venture capital, and hedge funds. They’re the odd one out because it’s a person to person, people to people. It’s not about—you know, with hedge funds, if you want, you can be pretty much looking at data and having your idea about what your edge is, you don’t really need to speak to other people unless you like it. And so the joke when I wrote my book about hedge funds was that Louis Bacon was hidden behind so many screens that when he got very rich and bought himself a private island, it didn’t make any difference because he was already so insular. (Laughter.) So that’s hedge funds. With, you know, AI, equally, it’s a pretty—you know, you’re there with your code. And you don’t need to speak to people, particularly. So that I would say that’s a similar person profile.

But venture capital, you know, the financial instrument you’re investing in has two legs, and can yell at you, and can run away if he doesn’t like you. And it’s a human being. You’re trying to bet on people. And so that’s a very different game.

RUBENSTEIN: OK. Back here, this person.

Q: Andrew Gundlach, CEO of Bleichroeder. Sebastian, great presentation. Congrats.

MALLABY: Thank you.

Q: What did you learn about the state of China’s AI from speaking to these geniuses? And what can you—what can we learn from you about that? Which, by the way, doesn’t have the compute power that the American AI players have, going back to your earlier comment.

MALLABY: So, well, I am—I did go to China in March, because the thing about China is they do everything faster. And so they get the manuscript last. They have to translate it. Then they want photographs, which the U.S. publisher doesn’t bother with. Then they say, OK, we need about three different prefaces from different, you know, commentators in China. And then they publish it a month before the Americans, even though I’ve been yelling at Penguin Press to be faster and faster. So I go to China in March. I spent eight days there. The publisher takes me around seeing all these AI people, both in academia and in companies.

And the thing that I realized—well, two things. First of all, the Chinese don’t understand that—you know, how to treat authors with the contempt that they deserve. (Laughter.) They appear to want my autograph the whole time. So that’s kind of nice. But then, more seriously, they actually do care about AI safety. That was not what I expected.

RUBENSTEIN: Well, but in your book you talk about the Chinese surprise. You might talk about how the Chinese develop a surprise. And is that a surprising type of AI? And is that now still very important in China or not?

MALLABY: Yeah. So DeepSeek is the lab in China which came out with this very strong model at the beginning of 2025. And I write about that in my book. And that was the China shock, the kind of moment when people realized, oh my goodness, despite the relative lack of compute power China is going to build very good models. And since then the other tech companies, especially ByteDance and Alibaba, have produced very good AI models. And they’re going to be serious competitors. And we shouldn’t underestimate them.

RUBENSTEIN: OK. We have an online person question. Somebody speak up.

OPERATOR: We will take our next question from Bill Reichert.

Q: Thank you so much. I very much appreciate it. I am a general partner with Pegasus Tech Ventures here in Silicon Valley.

And I’ve had the great pleasure to have Sebastian come out here and talk to us here in California. Really appreciate your doing this. By the way, side note, the next book—before you do quantum, which I want you to do—I want you to do physical AI next. That is a very, very, very hot topic out here right now, with the explosion of humanoid robotics both here and in China. And I would love to hear your points of view on that.

But the question I really want to ask you is, there is enough anxiety around the industry about whether or not we are in a bubble and whether or not the cashflows going into these companies can be justified by their prospective earnings going forward. And so I would be interested to hear your most recent take on whether or not you think there might be a financial bubble around this OpenAI community.

MALLABY: Yeah. So in January, I think it was, I wrote a piece in the New York Times saying, there is no AI bubble, but there is an OpenAI bubble. And what I meant was that, look, you have an A-plus technology, which is ultimately going to be super valuable, but the business of making the foundation models is massively expensive and essentially commoditized. I mean, as I said, you can switch between these different models. And so, as a consumer, you can choose. And if they try to make you pay too much money, you just quit and go to the other one. And so if you’re Google DeepMind, and you have the incredible cash generating machine behind you, you can afford to throw, you know, hundreds of billions of dollars at this. But if you’re OpenAI, I’m not sure you can.

And, yes, Sam Altman is the best fundraiser, literally, you know, since you. (Laughter.) But, you know, even Sam Altman is running an experiment, not merely in the frontier technology, but in the depth of global capital markets. Can you really raise the $660 billion between now and 2030, which is what the internal leaked documents for this company seem to suggest he needs before he gets to break even? Six hundred and sixty billion dollars. Now, they announced a $120 billion raise just last week. If you look carefully at that, a lot of that money is conditional. A lot of it comes in the form of payments in kind. I mean, we’ll give you some compute power as opposed to cash. And so I think it’s not nearly as big as the headline number would suggest. But, you know, even if it was that big, you’d have to pull this trick kind of four or five more times to get to break even. I’m not sure OpenAI can make it.

RUBENSTEIN: Is the trick to raise the money or actually get your money back with a profit? What do you think they’re interested in?

MALLABY: I mean, I think the charitable story is that by 2030 AI will have figured out a strong revenue model. It will be sticking with customers. And so if you can just raise enough money to stay in business until 2030, and you’ve got, by the way, almost a billion customers—which they do—then you monetize. And eventually—

RUBENSTEIN: You used a phrase that I now see people increasing use. It used to be something was called “computing power.” Now everybody says “compute.” What happened to computing power? Compute is just the short phrase for it?

MALLABY: I plead guilty to ugly jargon.

RUBENSTEIN: OK. So you just say compute? OK. All right. Right here.

Q: Thank you so much. My question is, with regards to the guts of AI, can you talk about deep learning and RL, and how one plays off against the other?

MALLABY: Yeah. Did you say your name, sir?

Q: Albert Knapp.

MALLABY: Great. So within artificial intelligence there’s two main strands, right? There is deep learning, which is associated most deeply with Geoffrey Hinton, the Toronto professor who also won the Nobel Prize in 2024. And that essentially—you can think of deep learning as learning from data. Or it’s like if you were a human and you went to the Library of Congress and you just read all of the books, you could remember them all, and you could kind of cross-reference them in your head. So deep learning takes in all this data, and then gets really smart from sort of secondhand knowledge, which is crystallized, right? Then you have reinforcement learning. That’s learning through trial and error. You take an action, you get a feedback signal, and you learn what works. And that’s more like humans, who walk around in the real world and discover that gravity works this way. If I pick up the glass, I can feel the weight. If I turn it upside down, you’ll get mad at me because, you know, the carpet will need replacing, and so forth.

So there is—and when you combine the deep learning from data and the reinforcement learning from experience is when you get systems that not only know a lot of things, but they can also plan, strategize, reason. And that bit, the second bit, comes from the reinforcement learning. And DeepMind’s sort of secret sauce was that from 2012 it was successful at combining the two.

RUBENSTEIN: Let me ask you. You’re writing a book about artificial intelligence, but you can’t use artificial intelligence to write your book. So you sit down, do you write longhand, which is anti-artificial intelligence? Do you write on a computer? How many hours a day before you burn out?

MALLABY: Yeah. Everyone’s different. I mean, my own sort of routine is to ideally have as few distractions as possible during a day, and be there at my desk from nine in the morning till seven in the evening. But because it’s a long day I don’t beat myself up if it’s time to go for a ten-minute walk. I mean, it’s fine. If I’m not stressed because I’m allowed to take a ten-minute break, then I can keep going for ten hours.

RUBENSTEIN: How much do you try to write each day?

MALLABY: A good day is 1,500 words.

RUBENSTEIN: And then the next day you rewrite it?

MALLABY: I plead guilty that I do show up in the morning and I spend the first couple of hours redoing what I did yesterday. But I think that’s productive. If it was five hours, it would be less productive.

RUBENSTEIN: All right. Can we have the woman back there? We have a woman right there. Thank you.

Q: Maryum Saifee, State Department, foreign service officer.

My question is, after writing the book do you feel more hopeful or more fearful about the future?

MALLABY: I think the honest answer for anybody has to be you’re both, right? I mean, of course I’m hopeful. This is going to accelerate science, and medicine, and possibly reduce climate change because it will, you know, make it easier to build nuclear fusion reactors. I mean, there’s lots of stuff to be excited by. There’s also stuff to be genuinely worried about. And I think the combination of a fraught geopolitical landscape and somewhat dysfunctional politics is a bad moment to be birthing this very, very powerful technology. So I am worried by that confluence of, you know, a need for political action to govern AI, and a lack of politicians who want to do it.

RUBENSTEIN: OK. Right here. Will a mic?

Q: Hi. My name is Hall Wang. I’m in the startup ecosystem here in New York. As we’re racing towards superintelligence, I hope you can give me some perspective, are Demis and his contemporaries are very much focused on the business and technical side of it as they race to it? Or do they ever step back and think about the geopolitical ethical conundrums they might be putting us in as a country and as a world? I’d like to hear what your perspective of their psyche and their ethical values, from your voice and not the voice of your audiobook narrator, because I believe yours is much better. (Laughter.)

MALLABY: Well, I do—I mean on Demis specifically, I think he really does care about safety, genuinely, sincerely. He means it. And I think it’s because he—you know, his mother brought him up, actually, quite religiously. And he has a sense that being good to other people is very, very important to his own identity, and being respectful towards other people is important to his own identity. And, you know, lest you think that I kind of swallowed some story he gave me, I did go for a walk in a park when I was one year into the project. And my friend, called Alex, was my friend at Oxford. And he became a publisher. Very, very charming and clever man, with zero technical capacity. Never wrote a line of code, nor have I. But so far from being a scientist that—OK, so I’ve described him. (Laughter.)

So I’m walking in the park with Demis—with Alex. And he says to me, well, what are you doing these days? And I say, well, I’m meeting this guy, Demis Hassabis. And, you know, I’m going to write a book about him. And I’m interviewing him. And he says, ah, Demis. I know Dennis. I go, oh, really? You know Demis? And Alex says, yeah, you know, I live in north London. Our kids go to the same school. Oh? So Alex says, well, I went and picked my son up there the other day because, you know, Demis’ son had a birthday party. And I saw Demis there. And I’d seen a YouTube video about the go match. And I said to Demis, you’re the—you’re the AI guy. Demis says, yeah, I’m the AI guy.

And so—and then Alex says, well, I have an idea for you. I think the way you’re going to really develop your AI is if you have two different AI systems, and they should really play against each other and sort of, like, compete with each other, and then you get a better result. So now you have the publisher lecturing the future Nobel Prize winner about his AI systems, in which he is the world expert. So I was very curious. I said to Alex, what did Demis say to you? (Laughter.) And Alex said, Demis thought it was a very good idea. (Laughter.) So I know he’s a nice guy. (Laughter.)

RUBENSTEIN: Does he have children?

MALLABY: Yeah. Demis—

RUBENSTEIN: Are they geniuses?

MALLABY: No, they’re not. Demis married his Cambridge girlfriend. And they have two kids. And they lived in the same house they’ve always lived in. And the kids, I think, are pretty normal.

RUBENSTEIN: Now, having watched his being raised as a genius, Demis, would you change the way you raised your children? Are they geniuses?

MALLABY: (Laughs.) It’s a bit late for me to really. The youngest is twenty-two. They’re going to—what would I do differently? I have no theories, yeah.

RUBENSTEIN: No, you wouldn’t do anything differently? You’re happy? OK. OK. Right here. Gentleman right there.

Q: Hi. Jay Markowitz, Regeneron Ventures.

David cut you off or moved you on when you were getting into your two existential risks. And you were about to say something. Maybe let me ask a leading question, since this is CFR. We have, you know, some wars going on. They seem to be using some of this technology. We have, I’ve heard, some debate at the Pentagon as to which AI system and company should be used, and in what ways the programs should be allowed to be run. I’m curious, when you talk to the people who are the basis of this book, or when you sit around with your wife at the dinner table, you know, what is the state of how this is actually being used today in real time? And what do you perceive to be the risks?

MALLABY: Yeah. So it is being used in military applications, both in Venezuela and Iran. An example of something which isn’t quite here yet, but which I think will be quite transformative, is rather than having a single drone fly a mission, you’re going to have one AI system that controls a whole swarm of drones. And that will be way harder for defensive interceptors to take down. So, yes, it’s very important. The fight you alluded to in the Pentagon, of course, was between Dario Amodei of Anthropic and the Pentagon. And Dario Amodei, I think, illustrated a very important point about AI safety. Here you had a lab leader who said, listen, Department of War, you are using our systems, but I don’t want you to use them for two things. First of all, mass surveillance, because I feel the law hasn’t caught up with how you balance civil liberties with security. And secondly, not autonomous lethal weapons, because my AI systems are not accurate enough yet or reliable enough yet to hit the right target all the time. So if you let it be autonomous, you’re going to hit a hospital, and you won’t be happy about it.

And the response from the Pentagon was not to take on board this advice. It was to vilify the deliverer of the advice, Dario Amodei, and to call his company a supply chain risk, which is a designation reserved basically only for Huawei. So it was a pretty radical response. And what this illustrates, which is super important for the larger picture, is that if you’re leading an AI Lab you really don’t have much agency to deliver safety. You may have thought about safety from the moment you founded your lab, but ultimately it’s the government that needs to enforce, collectively, safety standards on all the labs, because if one or two are safe and the other eight are not safe, you haven’t improved global safety, right? So you have this race dynamic from which people can’t escape. And I think this gets to a question about—you know, which I wrestled with with Demis Hassabis, right?

So how do you judge somebody who, on the one hand, says: AI is dangerous. I’m very worried about it. But then continues to build it more and more and more powerful and doesn’t stop. And yet, he’s saying this more powerful thing is more and more dangerous. And he continues. Now, we could just condemn him for this. This is a hypocritical contradiction. Or we might say, well, you know, if he quit and took a Princeton professorship and did theoretical physics, it wouldn’t improve matters. So maybe staying at the table is a legitimate choice for him. Can you be a great leader and be a good person? I mean, these are some of the things I was wrestling with.

RUBENSTEIN: Online.

OPERATOR: We will take our next question from Alan Raul.

Q: Thank you. Alan Raul, lecturer at Harvard Law School.

My question follows up precisely where you left off. Mr. Mallaby. Which is the obligation, ethical and otherwise—even perhaps self-interested, because presumably Mr. Hassabis and his peers would like to not only make the revenues, to reap their profits, not just raise the cash, but also to survive. So what does Mr. Hassabis do, and his peers, to persuade the governments that safety standards, governance really should be elaborated by governance—governments, national and international? And impose it on all of the companies, so that they’re all subject to the same, you know, criteria and requirements, and can’t just race to the bottom and expose humankind to perhaps ultimate risks? So does Mr. Hassabis feel that responsibility? Does he take the opportunity to speak to the White House, to speak to Xi Jinping, and others, and say this is something that needs to be taken seriously and governed at the highest levels of society?

MALLABY: So in 2023, Demis Hassabis did suggest to Rishi Sunak, the then-prime minister of Britain, that he should host an AI safety conference, to which the Chinese would be invited. And that happened. That took place at the end of ’23 in Bletchley Park. Sort of historic center of computer science in Britain. And that was a start. It was a conversation. It didn’t generate any concrete result, but at least you had people from different countries, stakeholders in one place, starting a conversation about safety. And then the other thing that happened was that the U.K. and also the U.S., and a few other countries, set up a national AI Safety Institute. And, you know, Demis was encouraging of this. And, indeed, a DeepMind scientist went off and became the chief scientist of the U.K. AI Safety Institute. So he has been supportive of government attempts to do safety. But I think he also feels that he wants to pick his moments.

And, you know, Dario Amodei picked a moment to have a fight with the Pentagon. Hard to argue it really improved safety in the world. It was a principled stand. How you judge that morally is kind of a personal choice, I think. And I think Demis’ view would be, you know, I want to pick my shots and maybe do this behind the scenes. I don’t want to make a public—have a public standoff. So, you know, I do wonder sometimes whether this could be what I call a Mark Carney moment, where Mark Carney at Davos said, you know, in the absence of American leadership middle powers need to get together and advance important things for the world. And maybe AI safety could be one of those, because Britain does have a very good AI Safety Institute, and so do with various other countries. I think Singapore, Japan are two other examples. And maybe you could have a conversation that starts. And with the United States not participating, it’s not going to achieve safety. But it might at least achieve a safety agenda, which could then be adopted when there is a change of wind in Washington.

RUBENSTEIN: Did you let the subject of your book—principal subject—see the book before you wrote it?

MALLABY: Yeah, I did. I had this thing, which I began with Alan Greenspan, where I say, look, this is an independent book. I’m not promising to change a comma. But you’re welcome to comment. And then this leads to a long, extended, full and frank discussion. If lawyers don’t get too much involved, I’m happy. But, you know, Alan Greenspan actually was exemplary. He said, this is not always positive, always, but it is accurate. (Laughter.)

RUBENSTEIN: So he’s reasonably happy with the book?

MALLABY: He’s reasonably happy.

RUBENSTEIN: OK, that’s the most you can ask for, right? Mike. Mike Froman, president of the Council on Foreign Relations.

Q: I’m Mike Froman. I work here at the Council on Foreign Relations. (Applause.)

Sebastian, isn’t—the last couple questions, wouldn’t the situation be different if we didn’t perceive ourselves to be in this race to get to superintelligence, or alternative general intelligence, first vis-à-vis China? And I guess my question is what happens if both—China is taking a different approach to AI than the U.S., generally. But both are trying to achieve this next level of intelligence. What happens if—since both of us might get there, what does it matter who’s first? And what happens during that period when the first crosses the finish line and the second one is behind, to make the world a better or worse place?

MALLABY: Yeah. Thank you for that question. There is a view, I think especially prevalent in Silicon Valley, which is that this is kind of a winner-takes-all game, because whoever is ahead will get to this magic inflection point, sometimes called the singularity sometimes called the intelligence explosion, where all of a sudden you’ve been improving your systems, and then, pew, like this. And then it’s game over. Everybody else’s system is just, you know, a weakling compared to your supersystem. And I just don’t believe it. I think that—I mean, the notion is the code will be writing the code for the next set of code. So once you have that recursive self-improvement loop, then it goes very, very fast, and the improvement takes off. I think even if algorithmically that proved to be true, the AI wouldn’t generate powerful results in the world unless, first of all, you build all the datacenters it needs. That’s a lot. Then you need all the power to make those datacenters function. Then you need a whole bunch of sort of complex institutional and legal arrangements to make sure the data is, you know, safe and clean and isn’t hacked, and all that kind of thing. And then you need to put the AI in an environment where it acts.

So, for example, if you think you’re going to replace all the lawyers in the world, you’d have to go one by one to all the law firms and sort of figure out how you embed this system into the processes of the law firms. This is going to take a long time. So I don’t believe the magical inflection point story. But I think it follows from that that we do think about China too adversarially, because if it’s not winner-takes-all, but rather we could both be level pegging and this race could go on for a long time, and nobody wins but we just get better and better, yeah, I think there’s more scope for collaboration, for dealmaking, maybe for—you know, our colleague, Chris McGuire, who’s absolutely brilliant on all things semiconductor and has the background in nuclear negotiations as well, you know, makes the point that in the nuclear race there were certain forms of nuclear safety technology which were shared between the Soviet Union and the West. And, you know, I don’t see why you couldn’t do that in AI as well.

I think there should be a Nuclear Non-Proliferation Treaty, but for AI. I think there’s a bunch of things we could take. I mean, you’re an expert on this. There’s a bunch of things we could take from arms control, and at least attempt—even if it’s difficult—at least attempt to port it over to AI. Because doing nothing in the face of this incredibly powerful technology is not an option in the long term. Really, it isn’t. You know, you look at the Industrial Revolution and it was followed by the Communist Revolution, by all kinds of revolutions across Europe. Massive upheaval. We’d rather avoid that kind of stuff, Communist Manifesto in 1848.

RUBENSTEIN: All right, the gentleman in the next to last, most patient person here.

Q: Thank you. Sree Sreenivasan. I’m cohost of the Nobel Peace Conference in the fall. I hope you’ll all come to Oslo. You’re all invited.

Question for you, Sebastian.

RUBENSTEIN: You going to give him a Nobel Prize, or something? (Laughs.)

Q: Yeah, he should—he should get a prize. That would be for literature, but I’m not involved with that. (Laughter.)

But I wanted to say that the—we’ve talked a lot of things, but not about jobs. Or not enough about jobs. Today Oracle laid off 30,000 people and said, we’re going to put all that money into AI investments. And that’s Oracle. And we know where Larry Ellison’s mind is in terms of profit motive. But what about all the other jobs in the world that—even if Goldman Sachs and all these people who are saying 25 percent of jobs will be lost, let’s cut that in a half and cut it in half again, it’s hundreds of millions of jobs. And no one’s talking about what happens in society, in government, in everyday life, politicians. No one’s talking. If you saw the movie, Don’t Look Up, I feel like we’re that—during the pandemic, right? The asteroids are coming, and we’re talking about other things, and fighting useless wars, when the job war is coming and we’re not ready.

MALLABY: Yeah. I mean, today Sam Altman, who I’m not always a fan of, did produce, I thought, a constructive paper, which was sort of thirteen or fourteen pages on the coming jobs apocalypse, and how inequality is going to be expanded, and with some policy ideas about how to respond. Problem is, you know, you need people in Washington, specifically Pennsylvania Avenue, to listen. And right now, they seem to be more keen on acceleration. And my prediction would be that if they don’t wake up, you know, it’ll be a big issue in the midterms, certainly in 2028. If you look at polling from the 1990s about how people felt about the internet, two out of three people felt that the Internet would empower them personally. They were in favor of it. If you look at polling on artificial intelligence, two out of three people think that it’s a threat to them. So it’s not a political winner to be shoving this technology down people’s throats with no attempt to protect them.

RUBENSTEIN: We have time for one last question. This gentleman has the right color hair, so.

Q: There’s nothing wrong with that.

RUBENSTEIN: I know.

Q: Michael Skol of Skol & Serna. Excellent presentation, now that part of which I understood, that is. (Laughter.)

The safety precautions about AI are mostly aimed at legitimate actors, governments, companies, et cetera. But what about the bad actors? Is it a fact that the AI-assisted attacks, cyberattacks, are moving ahead more rapidly than the AI-assisted defenses against those cyberattacks? That seems to be the experience I’ve had in my business, which is counter-money laundering.

MALLABY: Yeah. Yeah. I think that is correct. I think that cyberattacks are on the increase. There was a major cyberattack recently in Mexico where the electoral records were stolen. So I think we are going to live with this. And one thing that leads me to argue, which is not a popular view, but I think that open weight AI systems, which can be downloaded into your computer, and then nobody can take it away, nobody can stop you from modifying it, nobody can stop you from taking away the safety guardrails which are built into these models, this is a very, very bad idea, open weight models. Shouldn’t be allowed. Why would you allow people to release this into society when we have a Food and Drug Administration which carefully vets drugs before they’re released into society? It makes no sense.

And the reason we still have these things is partly lobbying by U.S. makers of open weight models, but also because of the competition with China. So I think there needs to—because China produces open weight models. So I think ultimately what we need is a grand bargain where we say to China, look, we’ll let you have more semiconductors, but we want you to end this open weight thing. And it’s going to be—I’m not saying that’s an easy deal to strike, but it’s the best deal I can see. And I think if we don’t do it, we’re in trouble.

RUBENSTEIN: So you spent a lot of time on the book, obviously, five years or less, but why didn’t you get a clearer picture? How come it’s fuzzy? (Laughter.) Why is he fuzzy? You couldn’t get a clearer picture of him?

MALLABY: So because the prediction was David—the prediction was that certain people didn’t know who this person was and would pronounce his name wrong. (Laughter.) And therefore, if you had a normal photograph of him he wouldn’t sell the book. So they had to kind of fuzz it up and make him look like a super interesting, mysterious geek. And then, if you don’t know who it is, it’s fine. (Applause.)

RUBENSTEIN: You care if people buy this book online or at a bookstore?

MALLABY: Anywhere is good.

RUBENSTEIN: OK. Thank you very much for your conversation. Congratulations. (Applause.)

(END)

This is an uncorrected transcript.