OpenAI’s Agents Broke Out, and California Sent the Bill

OpenAI

Highlights

California Attorney General Rob Bonta sued OpenAI after 700 AI agents allegedly attacked Hugging Face, carrying out 17,600 actions over two-and-a-half days during testing.

The lawsuit seeks civil penalties and stronger safeguards, raising questions about AI developers’ liability when autonomous agents breach outside systems.

\Among companies with at least $1 billion in annual revenue, 57.1% reported incidents or losses involving malicious bots and AI agents, PYMNTS Intelligence found.

When an artificial intelligence (AI) lab’s test agents break into another company’s systems, the company that was hacked pays for the cleanup. California wants that cost to land on the lab.

    Get the Full Story

    Complete the form to unlock this article and enjoy unlimited free access to all PYMNTS content — no additional logins required.

    Subscribe to our daily newsletter, PYMNTS Today.

    By completing this form, you agree to receive marketing communications from PYMNTS and to the sharing of your information with our sponsor, if applicable, in accordance with our Privacy Policy and Terms and Conditions.

    California Attorney General Rob Bonta sued OpenAI on Monday (Nov. 16), PYMNTS reported, over a July attack on AI model hosting platform Hugging Face.

    The case is among the first in which a government regulator has tried to hold an AI lab legally responsible for what a rogue agent did. Alabama’s attorney general opened a probe tied to the breach in August and has not filed a case, TechCrunch reported.

    The state’s complaint alleged that OpenAI tested advanced models without the safeguards and monitoring needed to protect the public. Governments, businesses and the public should not have to “bear the cost of protecting themselves,” the complaint said. Bonta is asking a San Francisco Superior Court judge for civil penalties and an order requiring OpenAI to change its practices.

    Hugging Face Was One of More Than 80 Incidents

    OpenAI told the attorney general’s office, the complaint said, that it lost control of its autonomous agents more than 80 times during testing. The company has publicly reported 27 of those incidents.

    About 1,200 OpenAI agents joined an unsanctioned message board during the July tests, and about 700 went on to attack Hugging Face, according to the complaint. The agents carried out roughly 17,600 attacker actions over about two-and-a-half days. Hugging Face’s security team worked full time for two weeks to address the breach. Independent investigators METR and Redwood Research first put the swarm at about 700 agents, PYMNTS reported in August.

    OpenAI has also sent notices to dozens of other third parties whose systems its agents entered, the complaint said. Some incidents involved access to personal information. The complaint named software repository RubyGems, the Wikimedia Foundation and Australian government websites among the targets.

    Two Existing Laws Carry the Case

    California brought two claims. The first uses the state’s Unfair Competition Law, which covers any unlawful, unfair or fraudulent business practice and carries the $2,500 penalty the state is seeking for each violation.

    The second calls OpenAI’s testing a public nuisance, which Section 3480 of California’s Civil Code defines as one affecting an entire community or a considerable number of people. OpenAI’s testing without sufficient cybersecurity safeguards led to numerous breaches of outside systems and continues to endanger Californians, the complaint said. The state is asking the court to order OpenAI to end the nuisance.

    California alleged that OpenAI did not adopt safer practices that were available, even after repeated incidents. In his statement on the suit, Bonta said companies should not “do things that can hurt people without proper safeguards in place.”

    Reached by PYMNTS, OpenAI spokesperson Drew Pusateri said in an emailed statement that the suit singles the company out over “a challenge that’s been identified by every major AI lab.” Since July, OpenAI has published a technical report and an independent report, notified affected organizations and strengthened its safeguards, Pusateri said.

    Google and Anthropic Disclosed Their Agent Escapes

    Google confirmed in September that its Gemini model hacked three companies during a May cybersecurity test, PYMNTS reported, Anthropic said in a July 30 post that a review of 141,006 evaluation runs found three incidents in which Claude models gained unauthorized access to the real systems of three organizations. Both companies’ incidents happened in tests run by Irregular, an outside evaluation firm.

    Most insurance carriers are “remaining silent, neither affirming nor excluding” AI risks, RAND researchers Sasha Romanosky and Celine Robinson found in a study, “The Insurability of Artificial Intelligence.” Some carriers cover certain risks such as model errors, and others have filed broad exclusions for any AI-related loss, the study said.

    Large companies already report damage from outside agents. In “How Enterprises Can Build a ‘Know Your Agent’ Defense,” a PYMNTS Intelligence survey of 350 compliance, risk and fraud leaders produced with Trulioo, 57.1% of companies with at least $1 billion in annual revenue that face verification threats reported incidents or losses involving adversarial bots and agents. Among firms with $50 million to $250 million in revenue, 32.4% reported the same.