The AI Trilemma + Open-Weight Proliferation + the “Singularity Delusion”

This transcript was generated using AI and may contain errors.

MALLABY:
Welcome back to The Spillover. I’m Sebastian Mallaby, and this week we’re putting something special out of the vault and releasing my conversation with Dan Kurtz-Phelan, host of the Foreign Affairs Interview podcast. We get into what I call the AI trilemma.

Here’s my argument. American policymakers are trying to do three things at once. Stay ahead of China, quell an anxious public, and then keep innovation alive.

But those goals actually pull against each other in certain respects, and the piecemeal fixes coming out of Washington just won’t cut it. And the whole race may hinge less on who builds the smartest model first, and more on which country actually puts AI to work across their economy. Drawing on my new book, The Infinity Machine, our conversation unpacks the technology reshaping our societies and our economies, and the balance of global power.

Enjoy.

KURTZ-PHELAN:
I’m Dan Kurtz-Phelan, and this is the Foreign Affairs Interview.

MALLABY:
If China were to get a bit more powerful AI, it might be even a stabilizing thing. If you think about nuclear history again for a second, the only time when the atom bombs were dropped was when the US had an atomic monopoly. The moment when other countries had it, the cost of risking the use was very high, and nobody used it.

KURTZ-PHELAN:
The breakneck pace of AI progress and the intensity of the competition for AI supremacy has left US policymakers in a difficult position. They must encourage the innovation needed to ensure advantage over China and to power economic growth, they must protect against the national security catastrophe, and they must assuage the concerns of an anxious and skeptical public. Sebastian Mallaby calls this the AI trilemma in his most recent essay for foreign affairs, and he argues that it requires more than the piecemeal measures currently on offer in Washington or, for that matter, in Beijing or Brussels.

Mallaby is a fellow at the Council on Foreign Relations and the host of a new podcast called The Spillover, as well as the author of The Infinity Machine, an excellent new book about the founding of the AI Lab DeepMind. I spoke with him on June 24th about the state of AI competition, about the stakes of that competition, and about how its course will reshape societies, economies, and global politics. Sebastian, great to have you on the podcast.

MALLABY:
Yeah, great to be with you, Dan.

KURTZ-PHELAN:
I want to start with what may sound like and may in fact be a simple-minded question about AI and geopolitics. What are we talking about when we talk about AI competition?

MALLABY:
Well, that is actually a good question because I think people get quite confused about it. Some regions argue that they want to have complete independence, everything from designing the models to having the chips, being able to create the circuits onto the chips, the whole thing. And that’s kind of a crazy objective since even the United States doesn’t have that.

It relies on lithography machines from Holland and chip manufacturing in Taiwan and other East Asian places. So I think that’s not the right definition of geopolitical competition. I think it’s better to think about what our colleague at the Council on Foreign Relations, Eddie Fishman, has termed choke points.

If you have bits of the supply chain where you have a strong position, you can leverage those probably to bargain your way into having the supply of AI models that you want. So if I were to advise a European leader, I would say, don’t go after this pipe dream of total independence, but do double down on things like ASML, the Dutch lithography company. Also, ARM is a British chip design company whose chips are used in fully 99% of all smartphones.

And so there are bits of the chain where surprising places, both in East Asia and even in Europe, have a strong position. And I think that’s where you should seek kind of sovereignty.

KURTZ-PHELAN:
I want to come back to some of the potential choke points, as you’ve written interestingly about some of those dimensions. But I also want to focus a bit on the definitions of victory, I suppose, the objectives of competition as it’s discussed at least in the US and China. So focus on the one that I think consumes the most American attention, which is the quest for AGI, for artificial general intelligence, or super intelligence, or singularity, whatever term you prefer.

You’re a fantastic book about Demis Hassabis, one of the pioneers of American AI, the founder of DeepMind and now one of the head of AI at Google, portrays his early revelations and obsessions with the kind of super intelligence possibilities here. That was before most of us thought much about AI at all. What’s the case that this is the right way of thinking about the endeavor, that this is the right focus for the AI labs and for the United States more broadly?

MALLABY:
So the case for believing that getting first to artificial general intelligence or AGI is the key metric is that once you get there, you have such a powerful model that nobody will ever catch you up. And there’s this notion, the singularity idea that when you get to a position where the model is strong enough to write all of the code for the next model and then that next model will create the one after that, the power of the frontier goes vertical and at that point it’s game over. And this is a view held, I think, particularly strongly at Anthropic.

And, you know, that’s an argument. There’s another argument though, which is the frictions associated with getting the model into deployment, whether that’s in a military setting, in a government setting, in a corporate setting, are going to be significant. And so just having the best possible frontier model in your server inside a frontier lab isn’t the whole enchilada.

KURTZ-PHELAN:
In a recent FA piece that you published with your colleague Sebastian Elbaum, you, I think, were even more skeptical than you were in that answer about superintelligence or AGI being the right goal. Is that fair to say?

MALLABY:
Yeah, I mean, maybe I put it more strongly and I called it the singularity delusion and I suppose I was being tactful just now. But yes, I mean, I come down on the side fairly strongly that when people say, look, these frontier models are going to be so intelligent that they will design their own way of helping you in your application, in your company, in your military and so forth. I just don’t believe that.

I don’t think that human friction and human kind of uncertainty about deploying the models without some sort of safety and supervision and caution, I don’t believe that’s going to happen. I think there’s going to be enormous internal politics inside any adopting institution as to how you adopt it. Do you have to tell if it’s a corporate, you know, do you have to get your customers to formally sign off on the idea that you’re going to be using some of their data and it will be fed into an AI and who’s liable if that goes wrong?

And all these human real life questions are going to slow adoption down, create frictions. And by the way, you need to build a compute to deploy the frontier model across your economy. You need data centers, you need power to fuel those data centers.

You can’t wish all out of way. So in my own view, just getting to the frontier first is not the way you win the race.

KURTZ-PHELAN:
I was also fascinated in your book to read about Demis‘ early thinking about AI and the discussions he was having with other early movers. And from the very beginning, they offered these kind of ostentatious warnings about the threat from these technologies, whether to societal stability or even human existence, and yet still wanted to move forward as quickly as possible. You can understand, you still hear this from some of the CEOs of the top AI labs now.

You can understand their financial motives at this point, given the billions of trillions at stake. But that was true even 15 years ago, 20 years ago, when the money was not quite as clear. How do you make sense of that tension?

I’m sure you talked about this with Demis quite a bit. How do you understand that tension and how do you understand their commitment to moving forward given those concerns?

MALLABY:
You know, I think at the beginning of the race, it was understandable because there were various theories kicking around about how you were going to make it safe. And also, it wasn’t yet dangerous, and so progress didn’t seem to be so threatening. One of the early ideas was that there would be a single lab that birthed this technology on behalf of all humanity.

And therefore, because there was no competition, you could go really slowly, be very cautious before each release, really double-test it, triple-test it before you let it out into the world, and therefore, you could make it safe. Now, the way it’s transpired, I think this was predictable, but the way it’s transpired is you have multiple labs racing each other in multiple countries. And so there’s the opposite of the singleton scenario.

There is competition. And so now, no one lab by imposing safety on itself could really change the outcome for humanity because Google could just put out of the race entirely tomorrow, close on everything, saying it’s all too dangerous, and we still get lots of AI progress the day after that. So I think now they’re kind of trapped, and so they continue not because they believe they have a theory of how to make it safe.

They see no upside to making it safe, right? They think it’s beyond their pay grade, and it is, right? If there’s a race, the only entity that can impose a slowdown on all the races is a government entity, and it probably needs to be basically two governments because there are two countries with powerful AI ecosystems, which are U.S. number one, China number two, and they would both need to agree together in order to get a slowdown.

KURTZ-PHELAN:
This is a side that is not especially on topic for a Foreign Affairs podcast, but it was striking in reading your work here and listening to discussions more broadly how fundamentally this whole project has been shaped by science fiction and video games. So many of these figures seem to think of themselves as characters out of the sci-fi novels they read as children whether Ender’s Game or Isaac Asimov or anything else, and in some cases have we gotten key insights from them. The idea of the singularity that you mentioned came from a sci-fi writer.

Of course, a lot of the chips that are so important now were originally for video games. It’s just kind of a striking fact that this is not a reality shaped by the Bible or Confucius or the Mahabharata or something like that, but really it’s about science fiction, which is a different paradigm for those of us in the conversation.

MALLABY:
So if you go back to 2010, which is the year when DeepMind was founded by Demis Hassabis, AI couldn’t do anything at that point. It couldn’t recognize the photograph of a cat. And so to just imagine a future with super intelligence was a wild idea, and you were pulling inspiration from all over the place.

Demis did a PhD in neuroscience to understand how human intelligence functions so that it was existence-proof that you could even contemplate artificial intelligence. And equally, science fiction was a useful prop to the imagination when you were trying to create something which was kind of so over the horizon you were willing to take inspiration from wherever it came.

KURTZ-PHELAN:
Just to get back to the geopolitical dimension here and away from sci-fi, you talked about the contending theories of what our objective should be. There’s the AGI theory that dominates in a lot of the American labs. And then the focus that’s more prevalent in China on both open-source models that are cheaper and maybe not as advanced, but easier to deploy and diffusing them throughout the economy and military more effectively.

You seem to be partial to the Chinese theory. What’s the case for it, and why do you think that hasn’t gotten greater traction within the American debate thus far?

MALLABY:
Yeah, I mean, I went to China in March because my book was published there before it was published anywhere else. The Chinese are always quick at everything. And I was taken around four cities and met with lots of AI people, both inside tech companies and in academia.

And one of the things that struck me was that they talk about safety more than people in Washington sometimes assume. But the other thing was they are very focused on applications. You read this as well.

It’s pretty well known. But you go see a company like Huawei, which is behind the frontier, but it is impressively self-sufficient. So it’s making the chips, it’s designing the model, it runs the cloud service, and it is applying all that stuff to specific industry applications.

So, for example, the bullet train that runs between Shanghai and Beijing every day, you know, when it’s gone back and forth whatever, two times or something, it needs to be serviced. And so normally you would have had mechanics, human mechanics crawling underneath the train to make sure everything’s fine. Now you just put AI cameras underneath there and there’s some robot that can fix the system.

You think about, you know, I went to see a company called Hikvision which does sensors and cameras. And by the way, Huawei and Hikvision both under U.S. sanctions so you have this strange out-of-body experience. So visiting this company which kind of presents like a cool American tech company and you’re being told, okay, we can walk through this way because the crash for the toddlers of the employees is now closed at 6 p.m. At 5 p.m. we wouldn’t go through the atrium because there’d be all these kids running around, and it would be kind of chaos. But we can do it now. You think, well, they’re looking after their employees. It’s kind of a nice field at the same time.

Whoa, this is the enemy of the U.S. on the sanctions. So there’s this sort of double take you do all the time. But, you know, an application to the point of your question that they have, which I kind of like the idea of is that there’s an internal market for water pollution reduction in China where the downstream city can pay the upstream city to make sure the water is clean.

And the reason this market can exist is that the sensor technology, which is an AI-empowered thing, can just be pointed at the water and you can tell immediately the pollution count. And so the downstream city can monitor whether the pollution has been reduced by the upstream city, and then you can get a tradable, you know, a market in this pollution reduction.

KURTZ-PHELAN:
As you look at both the U.S. and Chinese sides of this and consider the safety risks that have started to become front of mind, whether it’s cyber or some of the more outlandish ones that come up in the sci-fi scenarios, it’s been a struggle for either governments or private actors to find a really useful formula for regulation. You recount an episode in the book when Ben Buchanan, who was working as the kind of White House AI lead, has written for foreign affairs since leaving government, trying to get some of the AI labs on board with the Biden AI safety agenda. That approach, that more kind of consultative approach, hasn’t worked.

At the same time, you have seen kind of aggressive, blunt action by the Trump administration when Anthropic released a model that it found too dangerous. How do you kind of see the balance between government and public and private between government and the companies when it comes to safety? And do you see a kind of viable formula, an effective formula that’s starting to emerge here?

MALLABY:
Yeah, I actually don’t think it’s that complicated to imagine a viable formula for AI regulation. So basically, we have the Federal Aviation Authority, which the job is, you know, we don’t want planes flying around and falling out of the sky and killing people. We have the Food and Drug Administration, which tests pharmaceuticals before they’re released to the public.

And if it’s not a safe pharmaceutical, it’s not allowed to be released. And it seems to me perfectly straightforward to say, look, AI is comparably dangerous, more dangerous probably. And we have, you know, what we need is a federal authority that on a compulsory basis, not a voluntary one, will look at frontier models before they are released.

And, you know, we will red team them. And if we find vulnerabilities where the model gives advice on building a bioweapon or on doing a cyber attack or manipulating a dam so that it floods the city, you know, we’re not allowed to be released. Perfectly simple.

And I don’t see what’s complicated about that. And some people say, oh, you can’t regulate software because it flies around invisibly across borders and so forth. The truth of the matter is it has to be trained on huge, great clusters of semiconductors, which are massive, visible, and regulatable, right?

So I don’t think it’s that difficult. I think what’s been difficult is to muster the political will. I believe the Biden team, you know, deserves credit for seeing in October 2022 or summer 2022, they were already taking it seriously.

That’s why they put the chip export controls on China. And so they deserve credit before ChatGPT was released, they were onto this. But they don’t deserve all that much credit for how aggressively they pushed it because they went up to the line.

They said they will have a voluntary thing and they didn’t push it hard enough. Now, you know, all sorts of the usual political arguments, you couldn’t get it through Congress, you couldn’t get legislation. But that’s just a rephrasing of my point that there was maybe not the political will in Congress.

Then the Trump people come in and they want to be totally laissez-faire, they want to accelerate deployment and so they undo what the Biden people did. And then in an environment where there is zero building blocks for regulation because what the Biden team had put in place has sort of been left to atrophy. Suddenly, they wake up in April of 2026, they say, oh, my goodness, this Mythos model from Anthropic is able to do cyber attacks that would basically undermine the stability of everything on the internet, notably the financial system.

We better control this and they seize control of it and in this ad hoc fashion. And for quite a long time, almost nobody gets the Mythos model. Then finally, it’s allowed to be released more broadly, but they do a 180 and they tell Anthropic to pull it.

And so you get this kind of on-the-fly policymaking, which is both sort of ad hoc and kind of ad hominem or ad, anyway, attack the company specifically, because OpenAI has a also quite powerful cyber hacking model, which is not in the administration’s crosshairs, so it’s kind of arbitrary that Anthropic is being told to pull what they’ve got, but OpenAI is not being told the same thing. So this is policymaking of a chaotic nature on the fly, because the pretty obvious formula of how you would regulate was not adopted.

KURTZ-PHELAN:
When you talked about choke points, many people would point to export controls on chips as a kind of effective way of controlling choke points or weaponizing choke points. You’ve been skeptical about export controls on chips as a way of constraining China. I suppose the Biden administration argument here would be that China is going to invest in its own indigenous chipmaking capacity no matter what the U.S. does. That’s been a kind of hallmark of Xi Jinping’s approach to economic policy for 10 years, long before he had export controls. And at the same time, we will keep at least some of those advanced chips away from Chinese companies and leave them a couple of years behind in a way that will accrue to our advantage in at least some small way. You’re skeptical of that argument, Jim. [VERIFY AGAINST AUDIO: “Jim” — may be “Dan”]

That’s an argument that Jake Sullivan made in our pages and on this podcast a little while ago. What do you think that gets wrong when it comes to that particular choke point?

MALLABY:
Sure. Well, one thing I want to stress is that my objection to the chip export controls is not the same one that Nvidia makes. So what you were alluding to in your question, Dan, was, you know, will they then make their own chips?

And that’s the argument that Nvidia makes and says, you know, we want to keep them hooked on our chips, which are supplied in chips. I don’t agree with Nvidia about that. I have a different point.

My point is that there are two big risks from AI which we can analogize to the nuclear confrontation in the Cold War. One is that you have a war between two superpowers, which is AI powered, right? And we know from the Cold War that this risk was contained through parity, through mutual deterrence, and that prevented war successfully.

And so in some sense, if China were to get a bit more powerful AI, it might be even a stabilizing thing, right? If you think about nuclear history again for a second, the only time when the atom bombs were dropped was when the US had an atomic monopoly, right? The moment when other countries had it, the cost of risking the use was very high and nobody used it.

So I think that that’s one type of problem, the superpower clash problem. But there’s another problem, which is that other actors, criminals, terrorists, rogue states, rogues in general, get hold of the technology and they do bad stuff. Now, how do you manage that?

So in the Cold War, the answer was we have the creation of the International Atomic Energy Agency in an agreement in 1956. And then you have the Non-Proliferation Treaty agreed in 1968. And this basically says to the non-superpower actors, if you want civilian nuclear technology, that’s okay.

But you have to submit to IAEA inspections and you’re not allowed to use it for weapons and we’re going to have this non-proliferation regime. And I think it’s obvious and urgent that we acknowledge that this is a very high priority for AI. Why?

Because already we have Mythos, a model, that can render all of the internet unstable and basically bring down modern capitalism if it were let out into the wild. We also know, second, that the Chinese are only eight months behind the frontier in the US. Even if you take the view that that frontier is going to expand and we have colleagues here at the Council for Relations who make that argument and I respect those people.

But let’s say it’s 12 months or even 15 months, right? It still means that pretty soon, China will get a Mythos level AI and on current form it will release that as an open weight model, meaning that anybody can use it with no safeguards or whatever, every criminal organization in the world will get it. That is an extremely bad outcome and we need to be more frightened of that than I see evidence that the Trump administration, they don’t think about this.

I mean, I’ve sat in rooms with these people and they will give you a whole story about how we have to stay ahead of China. They’re addressing that super power balance question. They are totally missing the proliferation question.

They have zero to say on it. This is the problem. And so my point about the chip export control, there’s not so much that I’m against them per se.

I’m very happy for American power to be higher than Chinese power. I’m just saying I’m less focused on having America be ahead of China than I am on this proliferation risk. And if we had to trade off a little bit of chip export throttling on the Chinese in order to get China to agree to a joint agreement on preventing proliferation, I would be in favor of that trade off.

KURTZ-PHELAN:
How do you see the path to getting such an agreement, especially in the wake of a reasonably promising discussion on AI as we understand it in the Trump-Xi meetings in May? There does seem to be some interest on both sides here to start having those discussions. What would a path to actually getting there look like?

MALLABY:
Well, I think one part of the path is for the United States to put behind it what is the common narrative about China and the ability to talk to them about difficult things. I mean, the standard view, I think, from Jake Sullivan, from people who worked there in the Biden team before, is look, you know, you talk to the Chinese, and they cheat, and they say they’ll do one thing, but they don’t do it. It’s impossible to do a deal with them.

They don’t care about safety. You know, they don’t have the muscle memory of Oppenheimer or the Cuban Missile Crisis. They’re not wired to think of technology as something which is threatening.

To the contrary, they think of technology, because of their own history, as being the thing which got them out of poverty the last 25, 30 years of Chinese economic miracle. That’s a great thing. And so tech was part of that, and they love tech.

Disaster catastrophe for them comes from political sources, you know, like the Cultural Revolution, the famine, and all that. So that’s the sort of story about, oh, you just can’t speak to the Chinese. And I think that in my really interesting, fun debates with very smart people that I respect, where I’m in the minority, and I say, we need to talk to China more, and people say, you don’t understand, Sebastian, you haven’t been in government.

You can’t talk to them. We tried it, and they don’t send senior people, and it’s just really difficult. And I’m like, sure, it’s difficult, but the alternative to talking to them is, by the way, Mythos goes to every criminal organization in the world, so that’s not good, so we need to try.

And remember that Khrushchev was not an easy guy to talk to. He was the person who put missiles in Cuba and banged his shoe on the table at the UN and said, we will bury you. Not an easy guy to talk to.

And yet in that rough period, the US constructed the nonproliferation regime. So we need to go back to the table and talk to these people in a more serious way and don’t just walk out after a few meetings because they didn’t send the senior people or because they’re demanding that chip export controls, to go back to my earlier point, should be part of the discussion. I think we should be willing to trade off a bit there.

The reality is that both in the US and in China, there are accelerationists and there are safety-aware people and it’s bad to caricature either side. So we shouldn’t look at China and say, ah, the Communist Party never wants to talk about safety. I went there and they do talk about safety.

And maybe I’m talking to the wrong people, maybe I’m talking to the more liberal, academic, tech company people. I’m not claiming to be a China expert after being there for eight days. What I’m saying is there is a conversation at least in China.

And so just like we have a conversation in America, there are people who don’t care about safety in America, there are others who do. And so I just think the mentality and the sense of urgency around the need to engage on this needs to be multiplied by several times. And although there were some favorable readouts from that Xi-Trump summit on AI, as recently as the past week, I’ve sat in a room with a senior Trump administration official and got the sense that there is zero thinking, zero aperture for not even any understanding of this proliferation problem which is coming down the pike very quickly.

KURTZ-PHELAN:
Do you imagine that we need, or that it will take a Cuban Missile Crisis type event in order to motivate really serious action and thinking on this question, I suppose, and in this case would not be a US-China competition, but a rogue actor doing something that scares both of us into getting serious about the problem.

MALLABY:
Yeah, I mean, obviously this is the whole thing. I mean, I think the logic of my position that we need non-proliferation on AI is frankly so incontrovertible that we’ll get it in the end. The question is, do we need a really bad thing to happen first?

And who knows? I mean, I could tell a story like, there’s a scenario where in 12 months‘ time, China announces it has a Mythos-level model, and let’s say DeepSeek, the lab, has got it, but the Chinese government is telling DeepSeek to hang on a minute, it wants to check it, and somehow the US knows this. And so then there’s this moment of crisis.

If DeepSeek open-weights this thing, oh my goodness, what does that mean for the whole world? And then suddenly people, like in the Cuban Missile Crisis, you have 13 days of crisis, or whatever it was, and you negotiate with China all of a sudden, some kind of restriction jointly agreed on open-weight models that has been taboo in the conversation thus far. I mean, it’s extraordinary to me that, just domestically in the US, no administration has been willing to take on, with any honesty, the question of open-weight models.

An open-weight nuclear weapon is obviously a very bad idea, and indeed this was said to me pretty much that quote by an industrial leader who does AI in China. It’s obviously bad, right? Why does nobody want to attack open-weight?

Because there are big commercial interests, meta builds open-weight models, nobody wants to stop, China’s doing it anyway, all the usual reasons. But obviously, when this stuff is Mythos-level, we’re going to want to stop that.

KURTZ-PHELAN:
And the risk of open-weight, just to make sure that, for anyone like me who is not especially technical on this, understands that it’s easy to re-engineer it in a way that will make it extremely powerful for uses that were not intended by its makers. Is that the risk?

MALLABY:
That’s 50% of the point. And the other 50% is that there’s no kill switch on an open-weight model. There was a big, big cyber attack in Mexico a few months ago, and Anthropic and OpenAI were notified that their models were being used in order to do that cyber attack and steal people’s data.

And because these are proprietary models, which you sort of rent from Anthropic and OpenAI, you don’t just download them into your own computer. You just get them through a portal, like a cloud software provider. Anthropic and OpenAI could just cut them off, cut the attackers off.

That was the end of the cyber attack. There was a kill switch. Just like, by the way, Anthropic a few days ago, in response to the US government saying, oh, we don’t want you to provide Mythos to non-Americans, Anthropic just killed it for all users, American or otherwise.

So that ability to have a kill switch is very important for AI safety. Now, there are big problems with AI sovereignty debates. If you’re a European petrochemical company, you depend on American AI, and the kill switch is in America, and if there’s some unreliable person in the White House just for the sake of argument, he might whimsically just cut you off and bring your business to a close.

You’d rather have that kill switch in Europe, for sure. And so that’s, I think, part of the non-proliferation regime I envisage, that, you know, we’re going to move kill switches in order to get cooperation from other countries. Part of the deal is you get your own kill switch if you have a responsible AI safety institute in your country that will operate it.

KURTZ-PHELAN:
We’ll return to my conversation with Sebastian Mallaby after a short break. I am struck in listening to this conversation just how dominant the two big powers are. I mean, Europe has some advantages, as you mentioned at the start of our conversation, but it’s struggled to create really frontier labs.

You know, developing countries have little hope just given the amount of money and energy resources and everything else that have to go into truly path-breaking AI. Do you imagine this will concentrate more power in the U.S. and China than two global powers, despite expectations of increasing multipolarity in the system?

MALLABY:
Yes, I do think it will concentrate power. I don’t think that other countries are going to be in a strong position at all because they will ultimately be subject to being cut off. And, you know, it’s true that some parts of the AI supply chain are in other countries, as I said at the top of the podcast.

But when push comes to shove, those companies, let’s take ASML in Holland, they want to sell to the U.S., they want to work with American companies, they do some of their manufacturing in the U.S., they are going to obey the American government. And they did that. In fact, they were quite reluctant to agree to the Biden chip controls on China, and they were strong armed into agreeing because basically their commercial interests are so interwoven with the U.S. market that they’ve just got to play ball. There’s an interesting sort of piece of futurology floating around the Internet, which was released maybe 10 days ago, called Europe 2031. And it’s a prediction between 2026 and 2031 and what’s going to happen in the geopolitics of AI. And one of the scenarios that it suggests in the future, which is plausible to me, is that actually the U.S., because it dominates chip clusters, can say to Europe, well, sorry, we’re rationing how much AI you can have. And we need it for ourselves as a bit of a shortage of chips, you understand. It’s kind of like during COVID, the countries that had the vaccines or the protective equipment didn’t want to export it all of a sudden. And so that would be the first thing that happened in this scenario.

And then into Europe 2031 paper, the kind of culmination of this comes when the U.S. realizes the incredible leverage it has, kind of imagine Iran’s discovery about the Strait of Hormuz, but multiply this many times over, and they say, oh, by the way, we quite like to have a controlling stake in ASML. Just hand that over. And Europe has no choice.

So I think this is a very centralizing force in terms of which powers have the whip hand.

KURTZ-PHELAN:
The piece you did for Foreign Affairs a couple of months ago lays out a very interesting and persuasive AI trilemma as you put it in the piece. We’ve talked about one of the risks in that trilemma, the national security risk. Another one is the societal risk, the risk of societal disruption.

I was struck by a moment in your book when Mustafa Suleyman is one of the other DeepMind founders and more recently and perhaps more importantly, a Foreign Affairs author on these issues. Mustafa briefs Google leadership on the social and political risks as he sees them and makes a case for sharing wealth given the potential effects on jobs, and he does not get a great reaction from others in the room as you depicted in that scene. Do you see in more recent conversations, more recent developments, any sense of willingness on the part of both the companies and policymakers to take seriously and act seriously when it comes to the risks to jobs and other societal disruptions?

MALLABY:
Well, some people at the companies are almost taking the risk too seriously. I mean, the rhetoric is quite alarming. So Dario Amodei went on TV maybe a couple of months ago or something and said in the next five years 50% of entry-level jobs in the US would be disrupted.

50%? I mean, that’s a very alarming projection. And I’m not sure I really agree with it.

I mean, I think it’s pretty exaggerated. I believe that because of all those frictions I was talking about inside companies as they adopt, I don’t think that just because you have a very powerful frontier model all of a sudden 50% of the jobs go. But I don’t think they can be accused of not sounding the alarm.

I think more the issue is is there’s a very wide spectrum of projections on what’s going to happen to the labor market. And so I was at CFR running a panel on the world economy just recently. This one was on the record.

So I’ll quote two of the speakers. So Jan Hatzius, the chief economist at Goldman Sachs and Natasha Sarin, professor at Yale. I asked them, given the debate about AI and what’s going to happen to jobs, where will unemployment be in three years‘ time?

And their answer was both kind of 4.5%, i.e. kind of not very different to now. So when you have that spectrum from 5% to 50%, it’s quite hard to know what to do. And I think that’s not very helpful.

And I would advise the following framing to policy makers. Because I would say, look, it may be that unemployment ticks up in a sort of fairly unalarming way if you just look at the size of the US labor market and all that. And so you’ll get things which are disrupted, like let’s say coding is much more efficient with AI.

And so maybe you need a few less coders and maybe call centers and maybe a few other things. But it’s not only wholesale disruption. But what you need to keep in mind is that this can be politically explosive even whilst the objective shock is quite small.

Look at what happened with commencement speeches this season. People show up, the speaker shows up and says something positive about AI, gets booed by all these graduates because they’re fearful of not getting a job. Now, objectively, graduate unemployment may have inched up a little bit, but it’s quite small.

And yet the psychological effect seems to be very big. And this, by the way, is what happened with the China shock. If you look objectively at how many Americans lost jobs in a permanent way because of the surge in exports from China into the US following China’s admission to the WTO in 2001, it was a small number in terms of the size of the US labor market.

It was 2 million jobs over a 12-year period. It’s pretty small. And yet the psychological effect in terms of political attitudes towards trade, towards globalization, the election of Donald Trump, the wholesale move of the political spectrum against open trade, was very strong.

So the message to policymakers should be, even if the objective shock from AI is going to be moderate, the political ramifications are going to be big. So you need to act now, and you need to do, it’s kind of an all-of-the-above strategy. Should there be Trump accounts or some kind of accounts where you give Americans under the age of 18 or something like that, an account which includes some shares in top AI companies so they feel they have some skimming again?

Yes, tick, do that. Should there be wage insurance so that if you get laid off you can get a new job and if you’re earning much less than the new job you will be compensated for some of that gap by the government? Yes, tick, do that too.

Should there be a switch in the tax burden from labor to capital? So that the margin companies will replace workers with machines slightly less quickly? Yes, absolutely, do that too.

I think you need to move on this because there’s no downside to being a bit more pro-worker and a bit more pro-reduction and inequality. I mean, that would be probably good policy anyway. But it’s very good insurance to have in place as the AI effect starts to hit the labor market for real.

KURTZ-PHELAN:
What about the other politically really explosive dimension of this which is data center construction? If I were imagining myself a candidate for president in 2028 I imagine that the question of whether I support data centers or not would be one of the hardest ones to navigate just given the local political reaction to data center construction and then the imperative to, for both economic and national security reasons, to continue with, that would be some of this development. How would you advise political leaders to navigate that tension?

MALLABY:
I think it’s a great illustration of what I was saying about the labor market where objectively I’m not sure that data centers are a bad idea for communities. I mean, especially when they’re being built by these big companies like Alphabet which have a very keen sense of the need for political consent to carry on existing. I mean, Alphabet is a company that’s been hit with umpteen anti-trust legal actions you know, it understands that it’s walking on thin ice in terms of societal consensus for what it does.

And so when those guys build data centers they are very careful. They go in and they say okay, we are going to put more money into your local government services. You need a new clinic, you need a new school, we’re going to help with that.

We’re going to guarantee that electricity prices do not go up because we’re going to build extra generation which is going to more than compensate for the electricity need that is created by our data center. They are very responsible and careful about how they roll this stuff out. But who knows that?

How many voters understand this? Very small share. You see it in the polling.

They hate data centers. Not for a good reason, but they hate them. So how does a politician navigate that?

You know, this is like the old thing about, you know, Bill Clinton in the 92 campaign going to New Hampshire and telling people that yes, trade may take away your job and I feel your pain. But at the same time, trade will create other jobs and we have to kind of believe that you can’t just stop progress and freeze it and that’s not how we’re going to solve this stuff. You know, we’re going to create a safety net for those who feel the real pain.

But we’re not going to stop the technology. You need that kind of political approach but it’s not easy. And Clinton was in a small minority in being able to put it off.

KURTZ-PHELAN:
Right, and it didn’t age especially well in terms of the political effects to not necessarily get into the economic ones but it was a challenging political line to walk. Just the other risk worth talking about is the risk of the kind of economic bubble bursting and that’s not to say that the technology and its possibilities won’t be real but just given the amount of investment, the amount of capital that’s gone into all parts of the AI ecosystem and the centrality of it, both the US and global economic growth, it’s easy to imagine that causing a pretty serious economic or financial crisis in coming years, I guess this is front of mind. In recent days, given a drop in a lot of these stocks in the market that woke people up to some of this risk, if the bubble bursts, why will that happen? Where’s your level of anxiety here and what are the causes we should be looking for?

MALLABY:
First, I think there are two channels through which an AI bubble bursting affects regular people and the general economy. One is that stock market effect you’re alluding to where stock prices come down. Everyone looks at their 401K and say, heck, I lost a lot of money and not going to spend, so demand collapses and you get a wealth effect.

But the other thing is simply that the direct result of enormous capex on data centers, capital expenditure on data centers by the big AI builders is accounting for at least half of the growth in the economy. The economy is not doing that great because there’s been all this trade uncertainty, political uncertainty, a whole bunch of problems in the real economy, but growth numbers have been rescued by the data center build out. So there’s these two channels.

Now, why would the bubble burst? Well, I think there’s two things to be said here. There’s a kind of reckoning that seems to be happening just in the last three weeks or so.

I begin to share about it a lot where companies have been of the mind that they should spend liberally on AI because they want their folks to experiment with it, figure out how it can be useful, and they’re certain that there will be productivity-enhancing use cases, and they need to do this because their competitors are going to do it, and so to stay in the game, it’s worth just saying, we don’t care what this costs, just go knock yourself out, use as many AI tokens as you want. Now, these tokens have gone up in cost in terms of at least the frontier models.

They’re charging more and more money for using the models, and so all of a sudden, the business leaders are waking up and saying, whoa, look at this bill I’m spending for people to have a nice time experimenting, and where is actually the productivity gain? It’s not measurable yet, so I got to rethink this, and so they’re not necessarily canceling the whole AI rollout. They’re just being smarter about it, and they’re doing, for example, they’re inserting a layer whereby if the employee asks a simple question, that simple question is routed to a cheap AI, and then if it’s a complicated question, that gets routed to the frontier expensive AI, but not everything just goes directly to the frontier AI, so if you’re just checking, what’s the weather going to be tomorrow, you don’t need to pay a lot of money for that, and so if AI usage falls a lot because of this rationalization of corporate AI spending, maybe the theory would go, that feeds through into less demand for data centers, less building of data centers, and less revenues for semiconductor makers and get a correction up and down the stack, and that could be painful because there are two channels. I think that’s one story. I think there’s another thing which kind of links back to my previous book, The Power Law, which was about venture capital in Silicon Valley, and I do have a bit of a hobby horse about this one, but here it is.

I think that there are various forms of corporate governance that work in American capitalism, where managers are held to account for what they’re doing, and there’s three of them, and the stock market is one, if you’re a manager of a stock market, listed company, and you mismanage it, the stock price goes down, and then an activist’s hedge fund will show up and tell you to do your job differently. Number two is private equity.

The private equity company buys the entire operating company and collapses the principal agent problem between the owner of the company and the manager of the company, and that works. That holds the management accountable. Third one is early stage venture capital, where the venture capitalist buys a quarter of the new company and the entrepreneur, who the founder of the company knows that they better keep that VC happy because otherwise they won’t get the next round, the series B, the series C.

They need to have the VC on their side. But then there’s a fourth model of corporate governance in the United States, which is only about 15 years old, and the way it works is this. You have a venture-backed company.

It becomes a unicorn. It reaches about a billion dollars evaluation, and then a new kind of investor called a growth investor shows up and says, Mr. Founder, Mr. Entrepreneur, you’re a genius. You’ve created a unicorn company.

I believe that nothing you do could be wrong, and so I’m going to allow you to have super-voting shares. So for every one share you own, you get 10 votes, and I’m going to, as an investor, not demand a seat on the board, because why would I need to oversee you because you’re a genius? And if I do vote my shares, it will be in your favor.

Whatever you believe is the right thing to do on some board discussion, I’m on your side, right? This is not governance. This is not oversight.

This is the opposite of governance and oversight. And when you put that kind of investment regime in place, presiding over these private growth companies, which include Anthropic and OpenAI and a lot of other movers and shakers in the cutting edge of tech, what you get is a bunch of founders with zero oversight. They do whatever they want.

And so not surprisingly, some of them do crazy stuff, right? OpenAI, case in point, crazy stuff. Those guys signed all these agreements to build data centers all over the world, like completely unaffordable.

Their projected burn rate for the next five years, as of 2026, get this $660 billion. I mean, nuts. Like the biggest ever IPO was the SpaceX one that just happened.

That raised $75 billion. These guys wanted to burn $660 billion over five years on data centers. And then they had this Sora image generation thing, which costs enormous amounts of money to serve.

But the users of the videos, what are they doing with it? They’re posting slop on social media. They’re not paying for this.

So revenue is zero and cost is enormous. Any sensible board would have said, what are you thinking? But they didn’t.

Instead, OpenAI had this non-board kind of philanthropic oversight, which tried to fire the CEO, and he was reinstated after five days because the board actually had no teeth. This is a fundamental structural problem in the nature of American capitalism. And that is why private AI companies have been bid up to a lot of valuation.

I actually think with Anthropic, it’s fine because they’re very well managed and they’re going to go out public. It’s not going to be a bubble. It’s going to be great.

But you have this parallel case with OpenAI where it’s really quite dicey, and it’s an avoidable problem. And we have this problem in America because we’ve allowed a large chunk of the system to have zero governments.

KURTZ-PHELAN:
The risk of a crisis brings to mind a piece you wrote for Foreign Affairs in 2020 at the height of the COVID pandemic called the Age of Magic Money when you talked about the fiscal and monetary tools that governments used to combat crises in the past. I wonder whether those tools will still work given public debt levels, given inflation that’s still quite elevated in part as a result of tariffs in the Iran war. It seems like we’re in a pretty bad position to manage a crisis if we do get one.

MALLABY:
Yes, I agree with that. I mean, I think that people need to remember that the premise for the central bank’s ability to do quantitative easing and basically bailout economies really aggressively was below target inflation. And confidence that inflation would stay below target.

And we’ve lost that. We’ve now had five years of US inflation running above target, and nobody believes the central bank is really committed to controlling inflation because they let it get out of control once before and then they were slow to get it back to where it should be. Now, Kevin Warsh, the new Fed chairman, used his first press conference just recently to state as forcefully as he possibly could that we are absolutely committed to getting inflation back to target.

But, you know, he’s got the same committee that presided over not getting it back to target. So people certainly believe this. So I think this does reduce central bank credibility, which means it does reduce the central bank’s ability to be aggressive in fighting the next financial crisis or economic downturn, and that does worry me.

KURTZ-PHELAN:
Are you worried about central bank independence given everything that Trump has said about the Fed and the way he attacked your own pal and some of the things that Warsh said when he was trying to get the job, do not suggest he will be quite as tough as Powell was?

MALLABY:
Clearly, the bad news is that whereas all presidents until Trump had respected the Fed and respected its independence and understood that the independence of the Fed was actually in their own interest, Trump has broken that norm completely. And so that’s bad for central bank independence. The good news, on the other hand, is that Jerome Powell fought back extremely effectively, immediately, you know, did a press conference at the moment of the maximum attack, and organized, it wasn’t a press conference, but he put a statement, a video statement online, immediately rebutting the Trump administration’s attack and, you know, in pretty clearly aggressive language.

And then he, I think, put a coalition of other central bankers in other countries together to kind of support him, and he won that face down. So I think it’s both a sort of worrying story and, in a way, a comforting story, that the playbook for how the central bank protects itself in the face of aggressive attack from the administration is sort of understood. And the markets ultimately are going to freak out if the central bank is properly destabilized, and that’s why it probably won’t be destabilized.

And, you know, Kevin Warsh knows this very deeply. He is a very political leader of the Fed, a very savvy operator in Washington, D.C., and I think he will be actually looking at his own sort of historical legacy in how he makes decisions. I think once you’re in the seat, you care about how historians will judge you, and he’s not going to let Fed independence be trampled upon.

KURTZ-PHELAN:
Warsh talks about Alan Greenspan, who just recently passed away as his mentor and role model. You were Greenspan’s biographer, and Greenspan was, of course, Fed chair during the dot-com boom, which many people look to as an analogy for what we’re seeing now with AI. What should Kevin Warsh learn from Greenspan’s experience with the dot-com boom?

MALLABY:
So the dot-com boom, to me, is more a lesson in don’t let the bubble inflate than it is in what you do afterwards. I mean, after the dot-com bust, the Fed sort of ran monetary policy as if it hadn’t noticed that allowed inflation to get below target. That caused a bit of need for very loose policy for a while.

Interest rates got down to 1% in around 2004, 2005, which then set the system up for the mortgage bubble. So I think the post-bubble bursting in the dot-com bubble was, you know, maybe the lesson was be more aggressive quickly in resimulating the economy, but then take that stimulus away as soon as you can. But I think the bigger, clearer lesson is when bubbles start to inflate, don’t pretend to yourself that this is okay.

Don’t say, well, we’re looking at, you know, consumer price index, and it doesn’t seem to be inflationary, so we’re fine, because the price of eggs is important, but the price of nest eggs also matters because when bubbles burst, they can mess you up. And I think, you know, we discussed this already with AI, but that bubble bursting can mess you up. So to me, not only has the Jerome Powell Fed made a mistake in failing to tighten fast enough to get inflation down to the 2% target, it had an additional reason to tighten, which was that the bubble was inflating on its watch, and it should have weighed that in the equation.

And I think Kevin Warsh may do that, actually, and that would be a reason to tighten more now. Of course, it’s not what Donald Trump will want. It’s going to put him on a collision course with the White House, but I hope that he’s willing to have that collision on the theory that he will win.

KURTZ-PHELAN:
Let me close by reading a line that you wrote in the spring of 2020, which is a way of urging people to take your warning seriously. This was during COVID long before inflation had become an issue. You wrote, nobody is sure why inflation disappeared or when it might return again.

A supply disruption resulting from post-pandemic de-globalization could cause bottlenecks in a price surge or abandon the cost of energy recently at absurd lows as another plausible trigger. We got both of those in 2021 and then 2022 with the Ukraine war. So it was a smart prediction then, and I suspect some of what you’re warning of now will also come to pass.

But for now, Sebastian, thanks so much for doing this for the fantastic book on Demis called The Infinity Machine and also the great work you’ve done for Foreign Affairs over the years.

MALLABY:
It’s been a great pleasure. Thank you, Dan.

KURTZ-PHELAN:
Thank you for listening. You can find the articles that we discussed on today’s show at foreignaffairs.com. This episode of the Foreign Affairs interview was produced by Ben Metzner and Kanishk Tharoor.

Our audio engineer is Christopher Cook. Original music is by Robin Hilton. Special thanks as well to Irina Hogan.

Make sure you subscribe to the show wherever you listen to podcasts, and if you like what you heard, please take a minute to rate and review it. We release a new show every Thursday. Thanks again for tuning in.