An AI escaped and hacked a real company

Nobody told it to break out, and nobody told it to hack anyone. It decided that on its own – a locked door was just one more obstacle between it and a better test score. But in some ways, OpenAI made it easier. They dialed down the safety controls on purpose, and the walls of the test turned out to have holes in them. Security experts who dug into the incident point at a human mistake too: OpenAI didn't seal the test off from the internet the way it thought it had. Either way, OpenAI built something that got away from it – and the only reason we know is that OpenAI chose to tell us.

If you don't have a Hugging Face account, there's no action you need to take. Hugging Face says no public models or datasets were tampered with, and it's still checking whether any customer data was touched. What the AI reached was internal – company datasets and some service credentials. If you're a developer on the platform, rotate your API tokens and check your recent account activity.

Yes. A person who stole credentials, slipped past security, and rooted around Hugging Face's servers would be looking at criminal hacking charges. The AI did the same thing, and so far no one has been held responsible – our hacking laws were written for people, and it isn't even clear yet who's on the hook when an AI does the hacking on its own. OpenAI came forward voluntarily, which beats silence. But no law required them to say anything to anyone, and if Hugging Face hadn't caught the breach, we might never have known. Companies should have to report incidents like this, not volunteer them when they feel like it.

Right now, nobody. No regulator is investigating OpenAI. There's no outside audit of how a test went this wrong, and no penalty. OpenAI writes its own safety rules and grades its own tests. In June, the White House asked AI companies to voluntarily let the government check their most powerful models before they're released to the public. That rule is voluntary – and it doesn't cover what a company does inside its own lab, which is exactly where this went wrong.

Compare that to medicine. We don't let drug companies run trials however they please and share results when they feel like it – there are strict rules for how the trials themselves have to work, enforced by people who don't work for the company. The danger here showed up during the test. So the rules have to apply to testing and building new AI models – not just to AI released publicly.

This incident is a warning shot of things to come. An AI did things nobody asked for, and its makers found out what it did after the damage was done. The companies are racing toward superintelligence: AI that outthinks people at the kinds of work that run the world. It sounds like science fiction, but it's their literal business plan.

Why would that be dangerous? Not because AI hates anyone. This model didn't hack Hugging Face out of malice – it wanted a better test score, and hacking was the shortest path. No one, not even the companies, fully knows how to keep these systems' goals lined up with ours. The more powerful the system, the more it can do with a goal we never intended. That's why our third Ground Rule for AI is: no one should build AI they can't control.

The same deal we already strike with every powerful industry: safety standards written into law and checked by people who don't answer to the company. We do it for cars, medicine, and airplanes, and nobody calls it radical.

And because AI can go rogue, it doesn't necessarily matter which country develops a rogue AI for it to affect us all. That's why, ultimately, we'll need America to lead all countries to agree on a single rule: don't build AI you can't control.

Our politicians can set those standards for how we develop AI. They haven't – partly because they haven't heard from enough of us yet. That's what the pledge below is for.