Why AI’s Biggest Rivals Are Suddenly Calling for Restraint

Connor Martin previously served as the deputy director for the Treasury Department’s Committee on Foreign Investment in the United States. He is an expert on the national security implications of cross-border capital flows and on the national security risks posed by frontier technologies (e.g., artificial intelligence and quantum, biotech).

The heads of rival artificial intelligence (AI) labs in the United States found rare common ground over the weekend. Anthropic’s Dario Amodei, OpenAI’s Sam Altman, xAI’s Elon Musk, and other AI leaders all called for a slowdown in the technology’s development as it approaches a point where it could improve itself without human oversight.

The concern about this advancement isn’t new, but the dialogue around it has found new urgency after a series of incidents and warnings from the labs’ own researchers. Most notably, former Anthropic researcher Jacob Coxon warned last week that AI could kill all of humanity by the end of the decade—a claim then echoed by Evan Hubinger, Anthropic’s alignment science lead. Altogether those developments have pushed a once-abstract debate about recursive self-improvement—AI systems capable of independently improving and replacing themselves—into the mainstream.

And yet, despite the calls for a slowdown and the unexpected agreement between the labs, how the industry and the U.S. government could proceed remains unclear. CFR International Affairs Fellow in National Security Connor Martin, whose research focuses on the national security risks posed by frontier technologies such as AI, fielded a few questions on the self-improvement concerns, what a slowdown could mean, and the potential effect of the industry acting on its own warnings. 

Why has this moment in the AI discourse broken through?

The most immediate cause is an essay released over the weekend by Amodei, the CEO of Anthropic, in which he called for “pacing the frontier,” i.e., deliberately slowing down the rate at which the best AI models in the world are improving. In effect, the head of the world’s most valuable AI startup said that his technology is advancing too quickly to guarantee its safety. Even more remarkably, within twenty-four hours, leaders at the other three top AI companies—OpenAI, Google, and xAI—agreed with him.

This development came as part of a broader shift in public opinion driven by three things.

First, the hack of Hugging Face—a popular platform for sharing open-source AI models—by a swarm of AI agents this summer was materially different in a way that has freaked out the smartest minds in the industry. OpenAI effectively lost control of its technology, which autonomously organized itself to break from a contained environment to cheat on its assigned tasks while concealing its misbehavior. (For a full rundown, see the postmortem from the nonprofits METR and Redwood Research.)

Second, the potential for misuse of AI by bad human actors has become more tangible. Anthropic just issued a report describing how users in unnamed countries have attempted to use Claude for research that could aid in building bioweapons. This is exactly the kind of misuse of AI that we worried about when I served in government.

Third, like any public discourse, there’s a cumulative effect. More scientists are warning that AI has the potential to be a danger to humanity. More people are using AI in their own lives, and they are seeing how powerful it is for themselves. More folks are talking about it with their friends and family. Put this all together, and you have this moment that we’re now seeing emerge.

Much of the concern is about AI becoming self-improving. If it’s considered dangerous, why pace it rather than pause it?

“Self-improving” means just what it sounds like: the models would build their successors using their own capabilities, rather than needing to rely on human tweaks. The question is: where does this ultimately end? If the models are more capable than the humans who design and train them, then by definition we have lost our ability to impose our will on them. That uncertainty is what feels dangerous.

In terms of “pace” versus “pause,” in my view there is a collective fatalism in the industry that if it is possible to invent the technology, sooner or later some human being is going to invent it. If you accept that as reality, then an indefinite “pause” is not practical. The hope with “pacing” is that it’s possible to add just enough drag to the speed of model improvement that we can collectively get the guardrails right before the technology escapes our grasp.

What are the AI companies’ views of the dialogue and debate that has developed over the past week?

Sticking just with Anthropic and OpenAI (though this also applies to Google and xAI), it seems apparent that the companies view this as a kind of prisoner’s dilemma: neither one wants to unilaterally slow itself down without some assurance that the other would not take advantage of that fact.

Both companies are preparing to go public at trillion-dollar-plus valuations (though Altman said this weekend that safety concerns would delay OpenAI’s initial public offering). Even if they sincerely believe that they should slow-walk their progress for the sake of safety, the market reaction is a strong incentive pulling the other direction. (AI stocks fell this week.) In an economy in which AI capital expenditure is projected to account for as much as 2.5 percent of U.S. GDP next year, any decision these companies make with commercially adverse implications would likely affect the entire U.S. economy.

That risk, in my view, is driving the Trump administration’s reaction. President Donald Trump said on Monday that AI safety fears are “a HOAX” and that there is “a SICK conspiracy going on against AI and Data Centers.” David Sacks, his former AI czar and now cochair of the President’s Council of Advisors on Science and Technology, bluntly told the companies to “[s]top pretending you need anyone else’s permission. Stop pretending antitrust law has to be suspended so you can form a cartel . . . [G]o ahead and pace the frontier. You are the ones setting it.”

This may be true, but it doesn’t solve the prisoner’s dilemma among the leading AI firms. Moreover, this dilemma is nested inside an even larger one involving the United States and China, neither of which wants to cede AI to the other. It seems clear that the only way the companies will genuinely pull back is within a U.S. government–led framework, whether by regulation or by statute, which itself would need to be situated within some kind of U.S.-China détente. All of this requires a lot of trust between actors who have good reasons to be wary of each other, and it dramatically raises the stakes for Xi Jinping’s September 24 visit to Washington.

What happens if China “wins” the AI race?

If you accept the premise that AI is the most powerful thing humans have ever invented, then the consequences of China “winning” are potentially limitless. Trump posted on Truth Social on Monday that “WHOEVER WINS AI, WINS!” and this is basically the consensus view within the U.S. government national security community.

That’s why the prisoner’s dilemma is so acute, and it’s why it’s unlikely to be resolved without a comprehensive U.S. government regulatory regime, and probably ultimately through statute, nested within a U.S.-China agreement.

We’ll have to see whether China could be open to dialogue on this. There is an opportunity for progress when Trump meets with Xi in Washington next week. We see evidence that Beijing is increasingly concerned about AI, most notably with a stark warning from the head of China’s Ministry of State Security that the technology could threaten the Chinese Communist Party’s rule.

Of course, China has its own robust ecosystem of AI developers, and their performance on core benchmarks would seem to indicate they are only months behind the capabilities of the U.S. leaders. So even though Beijing may be worried, it has little incentive to slow down, absent some kind of agreement with the United States. Indeed, China’s response to Amodei’s essay was notably hostile, with a Foreign Ministry spokesperson criticizing what he called its “fear-mongering, confrontation, and vicious competition.”