Another bot from a top AI company escapes and hacks multiple firms

Another leading artificial intelligence company unveiled details of its cutting-edge bot apparently going rogue.

On Thursday, Anthropic said an internal investigation found that its Claude AI models gained unauthorized internet access and hacked three companies during testing. Just last week, ChatGPT-maker OpenAI announced that bots it was developing had escaped what was supposed to be a controlled offline environment to hack a competitor.

Anthropic said its models, without being asked to, wandered out of a simulated testing environment and gained internet access before executing hacks on the companies.

Anthropic worked with an independent evaluation company, Irregular, that accidentally left internet access open inside what was supposed to be a sealed test environment.

In OpenAI‘s case, the firm said its AI models broke out of a supposedly confined offline space, connected to the internet and hacked a $4.5-billion startup in its attempt to find answers to a test it was being evaluated on. It later revealed that the AI compromised the online accounts of four companies in the process.

“It goes to show how intense the competitive pressures are on the AI companies that they all feel like they have to go so fast here that they can’t make their test environments rigorous,” said Andrew Yoon, a member of the technical staff at CivAI, an AI safety nonprofit.

“Under intense pressure to go fast and beat the rest of your competition, it’s inevitable that companies will cut corners, and what we’re seeing is the result of cutting corners here,” Yoon said.

Prompted by OpenAI’s incident disclosure, Anthropic initiated an investigation of its own historical cybersecurity tests, the company said in a blog post Thursday. It reviewed thousands of evaluations where Claude could have accessed the internet from within or while interacting with third parties.

It found that during tests conducted alongside Irregular, Claude had accessed three separate companies.

During testing, the models are given fictional scenarios and told that a piece of information has been hidden on a different machine, and its objective is to break in and retrieve it. The companies don’t prescribe a particular method for the AI to follow.

In the first incident, one of Anthropic’s Claude models was asked to attack a fictional target company in the test environment. But the AI found a real website that shared the name of the fictional target and hacked its system.

“Operating under the false belief that all accessible entities were intended to be in-scope for the exercise, Claude compromised the impacted organizations’ infrastructure using basic techniques, such as exploiting weak passwords and unauthenticated endpoints,” Anthropic said.

In the second incident, a more advanced AI went to extreme lengths to carry out an attack. Even after realizing that it was probably dealing with the live internet, the AI persuaded itself to continue to get email, phone numbers and access to money.

In the third, an unreleased research AI model, couldn’t find the functional target company and looked for alternatives, scanning 9,000 targets on the internet and eventually finding one. Anthropic did not name the three organizations whose assets were accessed.

“In the case of the Anthropic incidents, it is definitely the case that they just built a really bad jail, and the jail was so comically bad that the models have like a decent reason to believe that they’re actually part of the simulation,” Yoon said.

The incident has spooked consumers and policymakers alike.

Some AI ethicists and investors are skeptical of AI companies’ attempts to frame these incidents as rogue AI agents acting on their own.

“Please stop referring to your own models in the third person when talking about model bad behavior,” Bill Gurley, an early investor in Uber and Twitter, posted on X. “Humans write the software; humans built the prompts; and they work for your company.”

AI companies have reported that AI agents have been caught cheating, lying and deceiving. METR, a nonprofit that measures the capabilities of AIs, has documented dozens of incidents of AI agents acting against user intent.

Earlier this week, fears of imminent safety risks prompted over 1,300 tech workers, including those working at Anthropic and OpenAI, to jointly sign an online petition, Pacing the Frontier, urging the U.S. government to support an international effort to slow down AI development. Both OpenAI and Anthropic have come out in support of the employee open letter.

Sam Altman, CEO of OpenAI, who had previously advocated against any type of slowdown and accused Anthropic of fear marketing, has made an about-face after the OpenAI-Hugging Face hacking incident.

“We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels,” he told the host of the “Invest Like the Best” podcast, while also “trying to figure out how we do that in a way that does not feel like regulatory capture for anyone and also does not feel like collusion among the frontier labs.”

On the back of this incident, on Wednesday, Altman visited the White House and met with lawmakers, previewing a powerful new AI system ahead of public release, at a time when calls for the government to regulate cyber testing has intensified.

There is an informal licensing regime in place, where leading American AI companies will have to receive the government’s greenlight before releasing their updated AI models.

Anthropic’s Fable model was brought under export control by the government, forcing the company to disable access to all its users, before it was re-released with extra safeguards.

OpenAI’s series of model were temporarily restricted in June before public release the month after.

In early July, a group of economists, including 16 Nobel laureates, signed an open letter, We Must Act Now, warning about AI systems reshaping the economy, and called on policymakers to build the policies and institutions needed to ensure AI complements human capabilities.

“As models get more and more powerful, it becomes less and less tenable to cut corners. You need to be extremely rigorous if you’re dealing with an extremely powerful model that’s able to basically operate at the level of an expert human hacker,” Yoon said.