Nvidia’s Hugging Face Deal Highlights the Dangers of Open-Weight Models

This transcript was generated using AI and may contain errors.

MALLABY:
So, Rebecca, it’s great to see you here in the studio. It’s been a while. I’ve got a story for you.

Okay. Okay, so I’m going to set the scene. I’m on holiday, on vacation, as you guys say, in Spain, but I’ve been invited to this amazing kind of semi-secret AI conference in San Francisco.

So I leave the vacation early. I go to the airport. I do like 15 hours in economy class, get to San Francisco.

I’m kind of zoomed out of my mind with sort of, you know, time change and so forth. I go to this conference. It’s like two and a half days of full-on tension between, you know, the models are just crazy, crazy strong now.

You can’t believe how strong they are. They’re crazy, crazy dangerous. You can’t believe how dangerous this is.

What do we do? We have no idea. This goes on for, you know, 48 hours of non-stop, right?

I get out of the conference. I have a couple of hours free before I need to be in the airport. So I have an introduction and I just kind of, you know, I go see this person.

I’ve never met this person before. He’s in an apartment in downtown San Francisco. And I’ve kind of forgotten what the meeting is about.

So I go up and, you know, he comes down. He takes me to this sort of penthouse kind of lounge in this apartment, right? And he’s a very nice guy.

He’s got kind of cool glasses and curly hair and an attractive smile. And he’s like, you know, joking around. And he says, anyway, I want to tell you about my company.

So I go, great, great. He said, okay, so imagine this. I have got a solution where you will never get a disease again.

You will never die. You can assume whatever appearance you would like. You can be in three places at once.

I’m going to upload you to the cloud. Yeah. And then you can live in the cloud forever, suffering no mortal jet lag problems, right?

And it’ll be perfect. So you no longer have a corpus. No, but you can be embodied in a robot.

Oh, good. And there could be multiple robots whose appearance you could change. Okay.

Right. And you could roam around in any kind of environment you could imagine, because these can be digitally simulated. So it’s way better than being on Earth.

And he says, look, there’s one drawback. I go, what’s the drawback? He says, we’ve got to slice you into very thin pieces.

The way the technology works is we take very thin slices of Sebastian, and we photograph them with a very high-tech microscope thing.

PATTERSON:
Yes.

MALLABY:
And we upload all these images of each cell, cell by cell by cell by cell. We reconstruct you into 3D inside the cloud, and then you come alive again. Well, not really alive, but...

PATTERSON:
Yeah. It’s a movie pitch?

MALLABY:
This is a, I mean... This is an actual company. Okay.

He says he’s got people signed up, waiting to go. As soon as it works, he’s done. As soon as it works?

He’s constructed already the brain of a fruit fly. Oh, I saw that. And published the journal paper in Nature.

That’s a proper magazine, right?

PATTERSON:
Yes, it is indeed.

MALLABY:
Okay. 1,000 X bigger than the fruit fly is the mouse. 1,000 times bigger than the mouse is the human being.

So what do you think? Sebastian sashimi?

PATTERSON:
I think not.

MALLABY:
Malibu carpaccio?

PATTERSON:
You know what? Call me a Luddite. I will embrace...

Is it Luddite-ism in this case? Because the idea of being sliced and I’m a robot in the cloud... When you started the story, I was thinking, okay, you can live forever, be whatever age you want.

I’m like, Vampire Lestat? No. Is it a movie pitch?

No. Is it some kind of sci-fi or maybe the man’s just crazy? But no, because I saw...

MALLABY:
I think you might have got the right solution at the end there.

PATTERSON:
But I saw the fruit fly article. So maybe he’s not totally crazy? Yeah, that’s right.

Or maybe the people who signed up to do this are crazy.

MALLABY:
The tension between is it crazy or is it not crazy? This is the essence of Silicon Valley. He embodied it.

I had a fantastic couple of hours and then I went to the airport.

PATTERSON:
I may be stuck on this story the rest of the day. We’re going to talk about what’s scary, crazy, crazy, and what could be great.

MALLABY:
At this moment, you’re supposed to say, I’m Rebecca Patterson.

PATTERSON:
All right, well, then I’ll say that. I’m Rebecca Patterson.

MALLABY:
And I’m Sebastian Mallaby. Welcome to The Spillover. Okay, so let’s get serious, Rebecca.

In this episode, what I suggest is we’ve had big news out of the AI business. Jensen Huang of NVIDIA has made an eye-wateringly high offer for the leading platform for open-weight models. That’s Hugging Face, French-American company.

I think we should discuss it.

PATTERSON:
I agree. And what I love about this conversation topic is this is not just an M&A story. This is a story about the future of AI and everything it means, from slicing you to fruit flies and way beyond.

MALLABY:
Well, and I’d make that point sort of at a high level this way, right? We talk about the split in the artificial intelligence world between U.S. AI and Chinese AI. We talk about the rivalry between the frontier model companies.

That’s Anthropic, OpenAI, Gemini, made by Google. But in some sense, the deepest, most important split is a different one. And that is between the Jensen Huang open-weight vision of the future of artificial intelligence on the one hand, and the proprietary closed-model vision, which is the one advanced by Anthropic and OpenAI and Google DeepMind and so on.

And that is, I think, the deepest split. And it comes to life in the form of this bid for Hugging Face.

PATTERSON:
Yeah. And I want to talk about all of that. Again, I don’t think this is just an M&A deal to discuss, although it’s a pretty interesting one.

But let’s start with the M&A deal and just explain how you said crazy earlier. It’s a little crazy. And then we’ll get into all these different spillovers from it.

And by the way, can I just say it’s nice to see you in person here in New York at the Council on Foreign Relations. It’s been a minute.

MALLABY:
It’s been a minute. I’m sorry I missed the great conversation you had with Vijay Vaitheeswaran and Natasha Sarin before that. Oh, no, sorry, after that.

Yeah, it’s good to be back here in person.

PATTERSON:
It’s nice to see you. I was starting to think you had become Where’s Waldo. All right, let’s talk about Hugging Face.

So I think people who tune into the spillover by now have a pretty good idea of what NVIDIA is. Biggest company in the world by market cap, world’s leading designer of AI semiconductor chips, what we will refer to as GPUs. Hugging Face may be a little less familiar.

MALLABY:
Yeah, I think people have heard of Hugging Face because of that scary breakout from Open AI where the agents hacked into Hugging Face and did all kinds of bad stuff. But they may not know what Hugging Face actually does. So could you explain that?

And while you’re at it, where does the name Hugging Face come from?

PATTERSON:
Oh, come on. I have not researched this. You’re being cute today.

You’re still jet lagged. I have not researched the origin of the Hugging Face name.

MALLABY:
All right.

PATTERSON:
So you researched it. So you tell us, where does Hugging Face come from?

MALLABY:
Turns out it’s taken from an emoji, the name, a smiley face. And I wrote this down here. It’s a smiley face with two small hands.

So not the scream. And it’s supposed to convey, quote, warmth, affection, support, and care. But I think you have to be a millennial to get it.

PATTERSON:
I’m thinking more Gen Z even.

MALLABY:
Either way, it’s not me. And even though you are way, way, way younger than me and eternally blonde.

PATTERSON:
Yes.

MALLABY:
Maybe it’s not you either.

PATTERSON:
No, I am a proud Gen Xer. We’re probably both in that bucket, I would guess. But anyway, let’s get back to what Hugging Face does and stop focusing on our age.

It’s just a number. So Hugging Face is basically a global library. And researchers, developers, companies can publish, share, collaborate on open source AI models for free.

And the analogy I like with this is Wikipedia, which I don’t love, quite frankly. But it was a big innovation when it happened. So Wikipedia hosts articles that are written collaboratively by people all around the world.

And they’re used by people all around the world. Hugging Face does the same thing for AI models. So people can use Hugging Face to collaborate, create, refine their models.

One researcher might create a model. Someone else might jump in and iterate on it, improve it, check it for bugs. But the bottom line is you basically have this free app store for AI models.

People can go in and say, oh, I like that pre-trained AI engine. I’m going to take a little bit of that home. And they can make text, audio, video, analyze things, whatever.

What’s crazy to me about Hugging Face, and it shows that I’m like, you’re on top of this. I try so hard to be on top of this. And I was still like, what?

Hugging Face already has over 3 million models to choose from.

MALLABY:
And it’s crazy. It’s a bit like what GitHub does for regular software.

PATTERSON:
Yeah. Yes, it is. It’s the GitHub of AI.

That’s a good one.

MALLABY:
Okay, so if Anthropic or OpenAI are kind of like Apple, in other words, they’re making closed, polished products, which they sell for a premium. Hugging Face is sort of a central library for free open source AI. But the question is, doing something for free doesn’t sound terribly profitable, Rebecca.

PATTERSON:
No.

MALLABY:
So why does Nvidia want to buy it?

PATTERSON:
Right. So I think the first thing we have to say is that Nvidia wants Hugging Face enough that it’s offering just about $13 billion for a free library. So there’s got to be a reason.

I don’t think Jensen Huang is dumb. So then why are they doing it? I think if you just look at Hugging Face’s revenue, it’s about, you know, the $13 billion price tag makes no sense.

No sense. Hugging Face makes a little bit of money by selling some premium services to its customers. But its annual run rate revenue is around $150 million.

So with a $13 billion, with a B, offer, Nvidia is offering a price-to-revenue multiple of about 86. So just to put that in context, you’re like, 86, what does that mean? When Microsoft bought GitHub in 2018, it paid 25 to 30x.

So the bottom line, 86 compared to 25, 30, it’s an incredible premium.

MALLABY:
So why does Nvidia want to do that?

PATTERSON:
You know, my sense without knowing all the granular details is that this is a longer-term play. It’s offensive and defensive. But on the offensive, Hugging Face serves about 18 million developers right now.

So it’s the dominant hub where open weight models are created and downloaded and deployed. And so if Nvidia can control the hub, it can integrate its own software tools into the models that are successful. And that means users who want to run those models will do so on Nvidia’s hardware.

So it won’t be chips designed by Nvidia’s rivals.

MALLABY:
Right. And we know that Nvidia is facing more of this competition on chips than it used to. So, you know, it has to play defense.

And Google launched a rival AI chip called the TPU. That was way back a decade ago. I remember this because it was incorporated into the AlphaGo system.

And in fact, this is the DeepMind system that defeated the world champion at Go.

PATTERSON:
Yeah.

MALLABY:
And the reason that system did so well in the last phase of its training was precisely because the TPU chip was introduced. So hardware is very important. But more recently, others have also designed their own AI chips.

So Amazon has one called the Trainium chip. Microsoft and Meta each have one. And while the big hyperscalers mostly use these chips for their internal AI work, or in some cases, they might make them available to customers through the cloud.

So Microsoft has Azure and Google has Google Cloud and so forth. You also have competitors of Nvidia, the so-called merchant GPU makers, that compete directly by just selling their AI chips to customers. And that best example would be AMD.

So Nvidia is still ahead. Right. But it’s definitely losing market share.

Competition is heating up. And just in the past month or so, we got another contender entering the market. That is OpenAI, which unveiled an inference chip called the Jalapeño.

Jalapeño. Jalapeño. OK, sorry.

PATTERSON:
All right. I mean, we got a hugging face. We have Jalapeño.

All right. Where is that coming from? Tell me what I need to know about our pepper friend here.

MALLABY:
Yeah, I think it’s basically just on the rebound from calling their main model ChatGPT. I mean, what a terrible name that was. Only an engineer could possibly come up with it or like it.

Fair. And that’s, in fact, exactly what happened. Oh.

You know, and so now they’ve clearly overcompensated. They brought in the marketing people. They’ve got branding out the wazoo.

And they’ve come up with, you know, the Jalapeño, Jalapeño name, whatever you call it. And the arrival of the potato, the arrival of the Jalapeño shows that the AI chip market is really hot.

PATTERSON:
Oh, that’s so painful, Sebastian. OK, this is why America had to be independent, just so we wouldn’t have to hear British puns like this one. I mean, you are a dad.

So maybe I can just call it a dad joke and we can get the cross-Atlantic tensions off the table here. But look, whatever the name is, as you say, it makes sense that NVIDIA would want to buy a major distribution channel. It wants to get a leg up on its competitors who are trying to catch up and take their market share.

You know, when I was back at JPMorgan a million years ago, I had a boss who wanted me to go back to school and do more statistics and econometrics. Yay. So I went to NYU and got my MBA.

And while I was working full time and pregnant, but.

MALLABY:
Were flex.

PATTERSON:
Yeah, it was not even very subtle, but I don’t care. So anyway, one of the professors there, and I remember as I was thinking about our conversation today, it just like popped in my head forward vertical integration. It’s right up there with the transflexor or something.

But basically what we’re talking about is a manufacturer gets a distributor to gain control of the customer relationship, secure shelf space, lock competitors out. That is what NVIDIA is doing. Luxottica is, I think, a good example of this.

So if you’ve probably heard of the name, but just for anyone who hasn’t, big fashion house makes Ray-Ban, Oakley, Chanel, Prada sunglasses. And starting in the 1990s, it acquired Sunglass Hut, which I’m sure we’ve all walked by in airports and in malls. And it got some other optical retailers.

It did trigger some antitrust concerns, but ultimately the deals went through. So they not only made the product, but they had the channels to distribute the product.

MALLABY:
Yeah, I guess it’s possible that those antitrust concerns might come back this time and actually block the NVIDIA bid for Hugging Face. Four years ago, NVIDIA did try to buy the British chip designer Arm, which has a very strong position in the smartphone market. And that was blocked for antitrust reasons.

So this time, with this intent to buy Hugging Face, there could be antitrust regulatory concerns in the US, in the EU, in the UK, or even China, because forward vertical integration is really just a fancy word for boxing your competitors out.

PATTERSON:
It is a fancy term. And I agree with you, there is a chance this might not happen, right? This is not a fait accompli.

Although I do wonder, with the current government regulatory structure in the US, at least on this side of the pond, people might be fine with it. But let’s talk a little bit more about NVIDIA’s logic for making the bid in the first place. So it’s not just boxing people out.

It’s not just freezing the jalapeños of the world.

MALLABY:
I can’t make these jokes, but you can.

PATTERSON:
Yeah, I can. That’s not fair.

MALLABY:
Anyway, look, yeah, I agree. NVIDIA isn’t just trying to capture a lot of the business of running the open weight AI models. It’s actually trying to boost the whole universe of open weight models at the expense of the proprietary ones.

And to me, this is the interesting part, right? So NVIDIA has a different vision of how the whole AI market should evolve.

PATTERSON:
Wow. Okay, that’s a big statement. A vision of how the entire AI market should evolve, especially coming from the world’s biggest company that has also become, in effect, the central banker to the AI industry.

I’m channeling my inner Jensen Huang today with my attire. I’m not sure if you picked up on that. I got my black leather jacket, my black shirt.

Yeah, I’m trying.

MALLABY:
He needs to start wearing a medallion, actually.

PATTERSON:
Yes, I think that would suit him. But yeah, I think we should go on. All right, so we’re doing this offensively.

We’re changing the entire industry.

MALLABY:
Look, since the launch of ChattyBT four years ago, a lot of the attention has been on the open AIs, the Anthropics, the Google DeepMinds, the firms that actually invent the AI models. That was the sort of sexy part of the story. And to the extent that people cared about NVIDIA, it was because NVIDIA was the maker of the picks and shovels that enabled the gold rush.

NVIDIA itself was not producing the gold. But I think what we’re really seeing with this bid for Hugging Face is that NVIDIA is rejecting that sort of second rank designation by supporting openweight AI models. What it’s doing is it’s aiming to turn the invention of AI models into a commodity where there’s lots of competition, not much differentiation, and so not much in the way of profits either.

And the idea is that the collaborative openweight models are basically going to do to proprietary models what Wikipedia did to the traditional encyclopedia business. Destroy the revenues and the business model. And if proprietary AI models can be commoditized in this way, the economic upside from AI will be captured elsewhere, not least by the hardware makers, foremost amongst them NVIDIA.

So in other words, NVIDIA’s vision is that the picks and shovels should end up being far more profitable than the gold. Okay.

PATTERSON:
So the openweight models and moving towards this and this commoditization, it does seem like there is a growing mass of support from Silicon Valley for this, right? I mean, Jensen Huang put out his first ever post on X this July. They are all doing these long policy statements lately.

Maybe they’re using AI to write them. Anyway, but it was called Openweights and American AI Leadership, but it was his big push for openweight. And he had a coalition of 25 signatories on that one, including companies like Meta, Microsoft, Mistral, Palantir, on and on.

And he wanted to go on the record opposing government restrictions on openweight, which I thought was pretty interesting. So let a thousand flowers bloom.

MALLABY:
I think it’s important to understand why he thought it was necessary to put out a statement defending openweight, because I think it brings into this discussion what the downside is, right? And I’ve said it before on the show, but just to restate quickly, with open models, first of all, they can be modified. You can download them and you can change what the guardrails are doing and so forth.

So safety guardrails designed to stop you from, for example, making a bioweapon, God forbid, can just be removed. That’s the first problem in terms of openweights and safety. And then the second thing is that openweight has no off switch.

So with a proprietary model, it is not handed over to you. It’s just sort of rented to you through a cloud-based portal called an API. And if you, as the user, start to ask questions about making bioweapons, then the model provider can spot that and just switch you off because you’re just renting it.

And the flow of the answers can just stop. And so you’ve got an off switch with proprietary models. You’ve got guardrails that cannot be modified.

And that makes it a lot safer.

PATTERSON:
Yeah, and I’m not going against what you’re saying here, but just to think about this kind of comprehensively, there’s also some positives from the openweight model, especially on the economic side. I think you and I would both agree that if you have an openweight model, if you commoditize these models, it’s going to bring the costs down. And so you can have more users around the world getting to use these models.

And so deployment of the models can increase and less inflation. So that’s a good thing. But there’s another question.

If the AI world is commoditizing, how does that flow through to the AI firm revenues? The second I hear the word commoditization, I think, oh, there goes your profits. And what happens to the stock market?

And what happens to the economy from there? So I think the world you’re talking about, it doesn’t necessarily mean everyone goes down, but it’s going to be this frantic race to figure out what the new winners are in an openweight AI world. And at a minimum, you’re going to have a lot of volatility in the markets as people are figuring that out.

One last one I’d throw in, just taking the other side of your argument, you mentioned with proprietary, you’ve got control, right? You can cut someone off. If you have an open model, you can adapt and control what you want to build.

So if you’re a bank or a pharmaceutical company or some other company with valuable proprietary data, if you have an openweight model, then you can adapt it. You have your own customized AI on your own computer system, on your own cloud, and you are certain that your intellectual property, or more certain at least, is not going to be leaking out.

MALLABY:
Absolutely. I mean, those are all true things. The disadvantage that you get in terms of safety from openweight, the flip side of that is the advantage of cheaper models for users and more control for users.

The first person I met who was running a business in America and running Chinese openweight models inside his business was a very successful proprietary hedge fund where A, they’ve got loads of their own computer scientists internally so they can adapt the model however they want. So the adaptability of the base model is very useful to them because they know what to do with it. And secondly, their data is extremely valuable and very proprietary.

So of course they don’t want it to leak out. And when you look at the news just recently about this mathematics prize, and there were two researchers who were using an open AI product to win this prize and they were almost done, and then all of a sudden it seems like maybe open AI had its hands in the intellectual cookie jar and basically stole the research they were doing because they could see these queries coming in. So that’s just a reminder of how it isn’t necessarily private if you upload stuff into queries and all that.

And so I think you’re right that there are some significant advantages in the openweight system as well as significant dangers.

PATTERSON:
Right. I mean, lower inflation is nice, but a global bioweapon is, yeah, that’s okay. So, look, we’re talking about some pretty serious spillovers, obviously.

Where do you come down on the trade-off, right? So if we have control, we have preventing intellectual property theft, we have inflation and economic benefit, and we have the end of the world. Do you want to see NVIDIA buy a hugging face or not?

MALLABY:
Okay, I just want to do, before I answer that, one sort of picking up on your investment observation, because I agree that there are lots of different risks floating around the AI boom story, and it’s useful to distinguish what these are. We talked on the show before about NVIDIA being the central bank of AI and extending all these financial guarantees to data center build-outs and so forth. And there’s a lot of this financial engineering creeping into the AI boom, and that worries people who think that the bubble is going to pop.

It’s too leveraged, it’s unstable and so forth. But I think in some ways what our discussion today is showing, there’s a whole different category of risk to the AI boom, which isn’t about financial engineering, but it’s really sort of a structural question of which companies get commoditized, which get the profits, as you were saying. And I’d add that the whole question about the shape of the AI industry, what’s commoditized, what isn’t, is compounded by other risks.

For example, if the frontier models are really scary and powerful, as my recent visit to San Francisco reminded me that they probably are, will the government just seize control over them? And then what happens to the economics of building the frontier model if the government is controlling which customer you can give it to? Or maybe alternatively, the sort of trickle of AI insiders, researchers who are saying this is too scary to go ahead with, maybe that trickle turns to a flood and then the frontier labs just have to pause because of a sort of internal revolt.

All these risks, right?

PATTERSON:
Yeah, and you could add, I mean, we could go on and on. I would add just quickly the data center question. I mean, that’s another risk to talk about.

This morning, a friend of mine was on CNBC and he runs an independent research shop and he mentioned that a U.S. senator this week had asked him what he thought the market and economic implications would be if 40 states banned data centers. I was like, oh, that was not on my bingo card.

MALLABY:
40 states, right?

PATTERSON:
And so, like you said, there’s the economic and market and those are real, but then there’s the backlash. What would it look like if suddenly you had all those stranded assets? And it’s possible, right?

We’re seeing this has become a bipartisan issue.

MALLABY:
Yeah, and so I guess my current feeling, I mean, all this stuff changes so fast.

PATTERSON:
Yes.

MALLABY:
One’s constantly sort of reexamining one’s bottom line, but for sort of people who are interested in the markets and investment side of this, it seems to me that there’s a lot of focus on this question of the financial engineering and the leverage and all that. And sure, that matters, but maybe actually a bigger risk if you’re an investor are these structural questions and political backlash possibilities and just like internal freak outs by the researchers. We don’t know how all that’s going to play out.

And maybe that’s actually more important in terms of an investment risk than the leverage stuff that we focus on.

PATTERSON:
I completely agree with you, but there’s a little part of me, Sebastian, that’s wondering if you’re avoiding my question. Do you think Nvidia buys Hugging Face? Do you think that deal goes through?

MALLABY:
Yeah, so I’m genuinely torn and I’ll explain why. So back in February, I wrote a essay, quite a long essay in foreign affairs with a co-author called Sebastian Elbaum, who is a academic computer scientist and the piece is called the AI trilemma. And we laid out some of the core trade-offs that you face when you’re designing AI regulation.

And on this particular question that you’re asking about whether an open weight future is a good future, what we came down arguing was open weight models are simply too dangerous to be allowed. They need to be restricted. We recognize the economic benefits of open weight.

We recognize that it will encourage faster AI adoption if you have open weight, but we just rank the safety concern higher. And so even though the risk of a proprietary AI oligopoly, right, because in a frontier model dominant world, there’s probably going to be two or three models that are just like, and then all of the profits and the power, you know, flow to them. That’s actually something I thought about more since I wrote that essay.

But even if I put that into the mix, I still think that safety is my top priority.

PATTERSON:
Okay, okay. And that’s interesting. My former employer, Bridgewater, the CIO and CEO came out recently with a big essay saying, we need, and this is a hedge fund, one of the biggest in the world that’s completely technology driven.

So they are leaning heavily into AI. They’re coming out saying it will hurt our business, but we need regulations. We need safety.

So there’s a lot of voices leaning your way. What did your co-author back from February, what did he say then? Do you know what he’s saying now?

MALLABY:
So that’s a great question. So the debate I had with Sebastian Elbaum at the time when we were doing the essay back at the beginning of the year, it was pretty interesting because to begin with, the other Sebastian had what I think is a classic view for an academic computer scientist, which is to say he thought that, you know, open weight in artificial intelligence is the natural extension of open source in traditional software.

And so open source has always been good. It’s always been the best way because you build a model, you put it out there for others to look at. They can see the code, they can improve the code, they can find bugs in your code.

And so that was the classic way to, you know, through the community, the human collaboration, the openness, kind of like Wikipedia, right? People edit each other’s stuff. So that was Sebastian Elbaum’s initial position.

And I still run into that position all the time when I talk to people from the tech community about what their views are on this. But as we continue to work on the essay, the other Sebastian’s perspective shifted. And the reason was that I think he realized that the traditional open source experience is actually not a great analogy for the new AI era.

So in the past, you built this great bug-free software through collaboration. But today, modern code review is not being done by humans. It’s being done by Mythos or other very good frontier models.

And even though we didn’t have Mythos yet, when we were writing this article, we had already pretty good models. And so the argument by analogy from traditional software just wasn’t relevant. And he kind of saw that, and that’s what made him change his position.

PATTERSON:
Okay, so as of today, both of you would say safety first, we need regulations. So you’re saying you wouldn’t want NVIDIA necessarily. I mean, it’s not that you’re against the deal per se.

It’s just this open weight world taking over is what gives you pause.

MALLABY:
Yeah, that’s right. And I think actually since the article back in February, the argument for restricting open weight has really gained momentum on balance. So again, the idea that human collaboration is the best way to get cybersecurity fixed is even weaker than it used to be because now we’ve got Mythos and Astra.

The idea that human collaboration is going to produce more capable models because you get lots of people collaborating together. The truth is that we’re going to get to model capability through the machines, the code, writing the new code. This is recursive self-improvement.

At Anthropic, something like 90% of the new code that gets merged into the model is written by the old model. So this idea of human collaboration is kind of anachronistic. So you put that together.

The upside from allowing open weight, open source collaboration is lower than it used to be. And at the same time, the risks of AI are becoming much clearer. We’ve talked already about that episode where the open AI agents hack out of their sandbox and go off and get into Hugging Face.

And we’ve now actually had the full postmortem on that. It’s even worse than we thought. 1,200 different agents collaborating together on secret message boards.

How wild is that? We’re not up against the Terminator, one bad machine. We’re up against like an AI swarm.

I mean, the risks are bigger. The upside of open weight is lower, and you sum it all up. And the old advantages of openness are just less compelling, and the risks are getting more serious.

And I’m more focused on containing the risks than on the problem of political and economic concentration that you might get in a proprietary AI world.

PATTERSON:
All right, it’s hard for me to argue with you, but I want to try to play devil’s advocate just for a minute. Your argument is founded partially on a safety failure that happened at open AI. But open AI is a proprietary lab, so a closed model.

So the closed model wasn’t protecting you from anything. It wasn’t an open weight lab. So what makes you so sure that proprietary labs, closed models, are going to deliver safer AI?

MALLABY:
Yeah, look, it’s a fair challenge. And actually, I had a long conversation with a cyber expert just a few weeks ago. And she had worked in the government and is now in the private sector.

But she was obviously a very expert and serious person. And she kind of made your point. She said, look, you know, over the years, being a cybersecurity expert, she looked at, you know, a company like Microsoft, that was the most proprietary, most resourced, richest of all the kind of proprietary software companies.

And famously, some of the code that they shipped had bugs in it. And it wasn’t always safe, right? So she’s just skeptical that proprietary labs can be trusted to deliver safe code.

And I guess my answer to that is that, yeah, I get the history, but I think AI is different. It’s partly what I said earlier about code review being automated. I kind of do believe in, I mean, Microsoft clearly didn’t always ship safe code.

But what about Anthropic? Well, Anthropic is going to put the code through a Mythos check. It has the internal model that finds the bugs.

So I’m kind of more inclined to believe that Anthropic will ship safe code than I would say Microsoft back in the day was likely to ship safe code.

PATTERSON:
With humans looking for bugs?

MALLABY:
Yeah, yeah, yeah. So it’s partly that. It’s also that AI changes the game between attackers and defenders in the sense that in the old world, you’d have attackers who figured out some new kind of malware, some new kind of phishing attack, whatever it was.

And they would go out there and they would successfully penetrate a bunch of different websites, cause a bunch of havoc. But the world couldn’t move on because they were doing this sort of manually, case by case. And so the defenders could kind of get smart and start defending themselves.

And yes, you suffered a bit of damage, but it wasn’t catastrophic. What I’m worried about is that in the new world, with automated AI driven attacks, when the bad guys get a bad model, they can just do like thousands a day. Right.

And the good guys don’t have time to kind of... And they’re free and they’re available.

PATTERSON:
Anybody can get one.

MALLABY:
Yeah, so what that points to is that, you know, we need AI regulation. We need the government to come in and say to the producers of the frontier models, you know, you have to... It’s like the Food and Drug Administration.

Before you’re allowed to put this out into the world, we need to be sure it’s safe. Yeah. And in that sense, again, the Microsoft history of shipping buggy software is irrelevant.

Microsoft was not being regulated when it shipped PowerPoint, right? I’m suggesting a new world in which Anthropic is regulated when it ships the next model.

PATTERSON:
Okay, that all makes sense to me. But earlier you were saying you’re torn. So what you just said right now doesn’t sound very torn.

It sounds pretty committed.

MALLABY:
Yeah, yeah. Yeah, I am torn. And the reason is that my worry is that although my preference for putting safety first is clear, I feel like we just may not do it.

And if we don’t do it, then it may actually be more damaging than helpful to be going on about restricting open weight. Because the fact is, you know, open weight does have some advantages, as we’ve been discussing. And so, you know, maybe the worst of all worlds is that the United States says, well, we want to start controlling open weight, so we do it internally.

And Gemma, which is the open weight model from Google, gets restricted. And Meta’s models are restricted. And that basically slows down deployment of AI in America, concentrates more economic power in the frontier labs, of which there are only three, basically.

And so you have these negative effects, but you don’t get more safety because the Chinese are still putting out their open weight models for any bad guy to use. And so that’s where I’m torn. Like, if you don’t think you can ever control open weight, maybe you shouldn’t even advocate for that.

PATTERSON:
So unless the US and China can agree on regulations, and we could say it’s all countries, but let’s just focus on the US and China, it’s a moot point.

MALLABY:
Yeah, I think that, yeah, exactly.

PATTERSON:
Okay, and this isn’t just, I’m digesting all that. This isn’t just about the US and China, right? So free open weight also means, and this is an interesting thing for geoeconomics, geopolitical balance of power, everyone can build and deploy.

And so if, you know, if we go back to where we started, and NVIDIA acquires Hugging Face, whether you agree with that or not depends on which AI future you want. If the merger happens, NVIDIA becomes even more powerful, juggernaut. And it will use that power to promote an open weight future.

So this deal pushes the entire world in one AI direction. So this deal is incredibly important, not just for a deal and for some banker fees. This deal will help shape the future of the AI world.

You know, given the US government right now, why wouldn’t it happen?

MALLABY:
I think, look, that’s a great summary. And I kind of agree with your bottom line, it probably won’t happen. And so do I reluctantly give up safety I would prefer and just sort of get real?

That’s where I’m torn.

PATTERSON:
Yeah, ditto. I mean, the Anthropic researcher who announced very publicly his resignation, because he’s concerned, and it’s certainly not just Anthropic, it’s all these companies, right? We’re seeing people leaving these companies, because they’re worried that we’re going to lose control of models.

So in an open weight world, that risk to me seems even higher. And while there might be economic benefits, there might be some interesting market opportunities, there’s also these existential risks that we just touched on lightly, but everyone knows are there.

MALLABY:
Yeah, I mean, you could say that, look, the Anthropic position is, we may lose control of the models. The open weight position is, we’re not even going to try to get control of the models. I mean, you just said earlier, you know, how many models are there on Hugging Face?

3 million? So it’s technology anarchy? It’s kind of, yeah, it’s, you know, you ship a lot of models, you put them on Hugging Face, the platform, people download them, they do what they want with them.

Nobody’s running the ship. Now, you know, in traditional software, that was probably fine. We never did think it was fine for pharmaceuticals, right?

We always wanted to regulate that. We never thought it was fine for aircraft flying in the sky. We don’t want them to collide with each other.

So we have air traffic control, and the FAA and all that. Now, I’m contending that with the arrival of AI, we have a shift where software now becomes like aircraft. Right.

It is existential.

PATTERSON:
It’s human lives. Yeah. Yeah.

Okay. This is a happy conversation. Hugging Face.

Yay, happy. What did you say? Calm, supportive, nurturing.

Yeah, all those things. But look, let’s just, before we finish up, let’s just, is there anything else we want to say about the differences between these two futures? I mean, like, different regions of the world.

I started going down a road of geopolitical balance of power earlier, and this was something I wrote about in my AI economic chapter for Hal Brands’s book a little while ago. If you have open weight models sitting outside the U.S. and China, open weight’s very attractive. It’s a way for a country in Europe that maybe is not developing frontier models to be able to adopt in a broad way, very quickly, very cheaply.

If you have open, U.S. or China, if they get mad at you over something, can’t just shut you down. So, in a future where businesses are increasingly dependent on AI, you don’t want your economy at risk of being shut off by the U.S. or China. So, having sovereignty around your AI, building a data center on your own soil using your own open weight models, you can control them, you can adapt them.

You’re not just renting them through an API. This could shift global balance of power.

MALLABY:
Yeah, and you can also add to the fact that for most countries in the world, deploying AI is the whole game. It’s not really about trying to make money from building the AI. I mean, there are some exceptions that we know about.

Taiwan with TSMC, obviously, is making money from the AI boom. The South Koreans have good memory chips. The Dutch have their lithography machines, ASML.

But for the most part, countries outside the top two or three are going to only make money from AI insofar as they deploy it successfully. And deployment is favored by the open weight future, where it’s easier to deploy.

PATTERSON:
Right, right, and a heck of a lot cheaper, i.e. free. I also assume open weight, you can deploy it faster. The data center build-out is going to be faster.

So, a triumph for NVIDIA could be a triumph for open weight, and that could mean globally stronger growth. If AI does increase productivity the way it’s hoped, right, and encourages more data center build-out, and more sovereign AI infrastructure build-out. And because the cost is lower, you know, that could be an interesting one, too.

I do wonder, though, even if the cost of open AI is significantly lower, if all these countries are saying, okay, cool, we can deploy fast, we need to put some money into this, do we still end up with inflation, higher yields, because everyone’s going to need the same materials for that infrastructure build-out?

MALLABY:
Yeah.

PATTERSON:
And then I go back to Fragile Four, right? U.S., U.K., France, Japan, memorize it, love it. Our very first episode, right?

If we have governments like these four and others who already have a lot of debt, and now are feeling, especially a country like France, right? It’s not building the models. So if it wants to play the AI game, it’s got to deploy fast and at scale, and that means it needs to make some investments.

But their bond yields are already trading above Italian bond yields. So the market’s already saying, we’re not so sure we’re happy about all of this spending, and that’s more spending.

MALLABY:
Yeah, I mean, going back to the Fragile Four, the U.S. is in a position of both getting the upside from this higher interest rate environment, because that’s partly reflecting the AI boom, and obviously the U.S. benefits from that. If you are Japan, there’s not much. I mean, there’s, I think, one of the main memory chip companies is in Japan.

But basically, they don’t have a lot of exposure to AI, and so they have the problem of higher interest rates without the benefit of the extra growth, and that’s where the pinch comes.

PATTERSON:
Yeah, yeah. Well, we could go on and on.

MALLABY:
So I already told a story at the top.

PATTERSON:
I loved your story.

MALLABY:
There’s a lot.

PATTERSON:
But you have a story this week. Well, it’s not a story. It was, and it wasn’t even this week.

It was probably a week and a half ago, but I’ve just, I can’t get it out of my head. So I love my Dyson vacuum. I love my Dyson hairdryer, so fast.

But the latest Dyson invention, I’m still wondering if this is a dud or who’s buying it. Is this the gift for someone who has everything? Dyson has invented, it’s not a toothbrush.

He calls it a, let’s see, hang on, I wrote it down because it’s an oral care device that combines sonic, I don’t even know what that means, sonic brushing, AI-powered gap targeting, liquid flossing, and mouthwash dispensing.

MALLABY:
Oh, right.

PATTERSON:
All of that, all of that for a mere $500. I buy toothbrushes for myself, for my family, and I was like, okay, what do I spend exactly on a toothbrush? A cheap one on Amazon is a dollar.

So instead of a dollar toothbrush- I can’t believe you’re already that cheap. Well, that’s like your Amazon basic. If you want to get like a fancy normal toothbrush, it’s probably somewhere around six or $8.

It ain’t $500, Sebastian. Anyway, the founder apparently has such a phobia of the dentist, so he thought if I invent this thing, I never have to go to the dentist again because my teeth will be perfectly clean.

MALLABY:
We can all relate to that.

PATTERSON:
Yeah, a hundred percent. But would it be as good as the actual dentist? I don’t know.

Anyway, all right. So people can check out the toothbrush and report back to us if they think it’s worth skipping the dentist.

MALLABY:
Send us an email about mouths, teeth, gums.

PATTERSON:
And with that... If you want to stay up to date on the latest episode of The Spillover and your oral health, sign up to receive an email alert anytime a new episode drops on cfr.org slash newsletters or click the link in our show notes. And if you have an idea or you just want to chat with us, please email podcast at cfr.org and make sure to include The Spillover in the subject line.

Our episode today was produced by Molly McAnany, Gabrielle Sierra, and our video editor is Claire Seaton. Our sound designer and audio engineer is Markus Zakaria. Our video producer and engineer is Justin Schuster and research for our episode was provided by the wonderful Liza Jacob.

You can subscribe to our show on Apple Podcasts, Spotify, YouTube, or wherever you feel like getting your podcasts.