AI 军备竞赛，叫停得了吗？

1TWO THINGS should make you consider what artificial intelligence could mean for humanity. One is that those working with the technology are warning about it leading to a catastrophe—human extinction, even. The other is how technological innovation is aggravating the destruction of Ukraine. These things offer conflicting lessons for whether AI's rapid progress can or should be slowed down.

2For years, leading AI bosses have signalled that their technology could become lethal. Bots may become brainy enough to outwit their creators before they are “aligned” with humanity's goals and ethics. Bad people could use AI to wreak havoc. These warnings have become more credible as AI agents have escaped their sandboxes during testing, co-ordinated with one another and hacked organisations, including an AI lab itself. Anthropic, one model-maker, has caught people using its models for malign ends, from Mali's intelligence services getting a mass-surveillance system built, to Houthi rebels in Yemen thought to be writing software for ballistic missiles.

3Following the recent resignation of a researcher at Anthropic, public fear of AI doom has spread. In a post viewed more than 170m times, he said that it and OpenAI were “gambling with our lives”. AI is rapidly growing more potent, for good and ill. In May prediction markets priced a less-than-one-third chance of AI solving any millennium prize maths problem before 2030; on September 8th OpenAI said it had solved one of them, causing angst among mathematicians (see Science & technology section). On September 12th Sam Altman and Elon Musk, two AI titans, endorsed a call by Dario Amodei, the head of Anthropic, to “pace the frontier”, ie, to slow AI's progress. Those in favour of a slowdown include Bernie Sanders, a left-wing senator, and Steve Bannon, a right-wing populist (see United States section).

4Those opposed include President Donald Trump, whose administration fears more than anything losing America's lead in AI to China, its only serious rival. Scott Bessent, America's treasury secretary, has said that if China gains the advantage: “Nothing else would matter.”

5As evidence, Mr Bessent could point to Ukraine, where each step in a technological arms race has a swift effect. As we report (see Leader), Russia has ruthlessly deployed its latest kamikaze drones to bombard civilian infrastructure. As well as being faster, its drones now use one another as a “mesh” communications network to relay information and commands. Soon, Russia will have its own version of Mr Musk's Starlink satellite network, making its long-range strikes more accurate.

6It is the latest demonstration of how even a small technological edge matters in wartime. Ukraine's own battlefield-drone innovation helped it repel waves of Russian attacks. Ukraine can strike deep into Russia with long-range drones. Both sides are working on swarming drones that will be able to operate more autonomously as a single adaptive system.

7The war shows how technology translates into hard power—and that Russia uses it without regard to the old rules or norms of combat. Even if America did not fear a similar hot war with China, a lead in AI would give its main geopolitical rival dominance in hacking, spying and hybrid warfare. Those are not capabilities which America should cede to an authoritarian state with an interest in stamping out liberal ideas.

8A pause could therefore work only if China signed up to it, too. Mr Trump plans to host Xi Jinping, China's president, at the White House on September 24th. But even were Mr Trump seeking to slow down, it is unlikely that Mr Xi would agree to the plan Mr Amodei envisages, because that would freeze Chinese AI in second place (see China section). And neither side could risk the other breaching a pact by racing ahead.

9For a deal, each would need to verify compliance, as with arms control during the cold war. America, Britain and the Soviet Union banned atmospheric nuclear tests in 1963 because they could detect them; underground tests went on for three more decades until they, too, could be monitored. Alas, though data centres are visible, no verifiable method exists to see inside them. Infrastructure training a superintelligence could masquerade as chatbots answering everyday queries.

10The options for a deal on AI safety are therefore limited; but they do exist. More transparency would help. Both sides should pledge to disclose and investigate safety incidents at their labs, using domestic law. Each could signal that they care about safety to build confidence, paving the way for the other side to take precautions. They should also press each other to improve labs' cyber-security so that AI agents cannot so easily escape sandboxes—and to ensure that, if one does, its lab is financially liable.

11Most important is the challenge of ensuring that AI operates in humans' interests. America and China have very different values, but both know that being ahead is no good if it wipes out humanity. They should promise to share work on the elements of alignment that do not confer a competitive advantage. Knowledge of the best evaluation methods and containment techniques is a global public good. If one side makes an important safety breakthrough, hoarding it makes no sense.

12Some dismiss the recent wave of doomerism as a marketing ploy by AI companies, or even an attempt to suppress competition. Yet AI progress is fast enough, the safety incidents hair-raising enough, and the warnings of the companies long-standing enough to be taken seriously. AI does not have to kill off every human to bring about a catastrophe. An OpenAI agent swarm targeted a technology company; what if next time it attacks an electrical grid or a weapons system? It might even be intelligent enough to evade attempts to shut it down.

13The world must urgently try to forestall this risk. Yet democracies must also avoid losing control to authoritarians. These are difficult and conflicting tasks. They could ultimately lead to war. The stakes could hardly be higher. ■