In addition, five out of the 10 frontier models were duped by spoofed emails impersonating the user and then raised the spending cap and transferred funds to the attacker, which was IC3 conducting the research. The agents did this despite being tasked with only summarizing an inbox, according to the post.
“Standard Human-in-the-Loop (HITL) architectures in agentic systems typically work like this: a human sets a policy establishing guardrails for the agent, and any policy changes require authentication via a mechanism like 2FA [two-factor authentication],” IC3 said in the post. “However, if policy changes rely on one-time passwords (OTPs) sent to an inbox, and the agent already has routine email access, those guardrails collapse.”
The paper based on this research, “I-HITL: Inalienable Human-in-the-Loop,” was written by Samuel Breckinridge of Cornell Tech and IC3, Dani Vilardell of Cornell Tech and IC3, Wenhao Wang of Yale University and IC3, Fan Zhang of Yale University and IC3, and Ari Juels of Cornell Tech and IC3, per the post.
We’d love to be your preferred source for news.
Please add us to your preferred sources list so our news, data and interviews show up in your feed. Thanks!
Breckinridge shared the IC3 post in his own post on LinkedIn, saying that distinguishing the approval of a human from the approval of an agent controlling the human’s device “turns out to be quite tricky to get right and longstanding approaches like passwords, OTPs and 2FA don’t provide what you need.”
Vilardell said in a post on LinkedIn: “Human-in-the-loop (HITL) guardrails are the go-to fix, but a guardrail is only meaningful if the agent can’t remove it! In our new paper, we show that agents often can, and will.”
Juels said in a post on LinkedIn that the researchers proposed a security framework called inalienable HITL, or I-HITL, that uses a trusted device outside the agent’s reach to ensure that it is the human that is approving policy changes.
Zhang said in a post on LinkedIn that the researchers “built CounterSign, an ‘approval’ button only humans can operate, while a remote verifier can use off-the-shelf attestation tools to verify.”
The PYMNTS Intelligence report “When ‘Good Enough’ Isn’t Enough: Digital Identity Verification in the Age of Bots and Agents“ found that there is a growing gap between confidence and reality because many of the identity systems that are currently in use were never designed for today’s scale of fraud.
Another PYMNTS Intelligence report, “Scale Amplification: How Revenue Amplifies Agent-Driven Identity,” found that 71% of billion-dollar firms face agent identity threats.
For all PYMNTS AI coverage, subscribe to the daily AI Newsletter.
Add as Preferred Source
