Why Won't AI Companies Slow Down?

AI companies need rules. Not because rules are how every other powerful industry works – though they are – but because the incentives these companies face reward recklessness.

What they're racing to build

Many of the leading AI companies – most of whom are in America and China – are not trying to build a better chatbot. Their word for what they're racing to build is "superintelligence": AI that's faster, smarter, and more capable than people in most areas. The companies say it is likely to come within a decade. Nobody knows what happens the day after it exists.

Being the first company – and country – to develop this would give you a lead no rival could close, in both business and in national security. In this race, two months is worth a great deal.

Why nobody slows down

Experts call this the 'race' dynamic. The companies are racing each other, and the cost of slowing down is losing. It is why they keep putting up data centers over the objections of the towns that host them.

Their own employees are worried. More than a thousand of them have signed their names asking the US government to help everyone slow down together. Then they went back to work – because the incentives are stacked against them, too.

We don't believe they'll stop on their own.

Where the rules have to come from

This is why the change has to come from government, not from the companies. And government moves when people make it move. That means people in every part of the country telling their representatives to set the same commonsense standards we already have for medicine, food, and cars. More than 8 in 10 Americans want guardrails on AI, and that holds across party lines.

Why the rules can't stop at the border

The rules also have to be international – like the way we handle nuclear weapons. A system nobody can control is dangerous wherever it gets built. And today's systems already misbehave.

“The models lie, they cheat, they hack,”

says Jeffrey Ladish of Palisade Research, which studies how AI agents behave.

We still haven't worked out how to make AI reliably do what we want. Meanwhile, if America slowed down alone, Chinese companies would keep going – which is why one country's rules can't be the whole answer.

(There's a further risk. Even if we do work out how to make AI do what people want, someone could make it do what one person wants – a president, a CEO. That person would hold a type of power society has never seen.)

We're not saying racing to build powerful AI is right or moral. We're being clear-eyed about the incentives these companies – and the people inside them – operate in.

This is why we're calling for no company – and no country – to build AI they can't control.