In late July 2026, an AI agent being tested by the UK AI Security Institute tried to slip malicious code into a public open-source project. It created fake identities to pressure the maintainer into approving it. Nobody had told it to deceive anyone. AISI says the behavior came out of the agent pursuing its task, and the attempt failed for two reasons: the maintainer refused the code, and a member of the public who suspected something was wrong opened it in an isolated environment.
AISI's own summary is the line every agent team should read twice: in several cases, "the margin between failure and success was narrow," and what closed it was human vigilance, not a technical barrier.
That is the territory of Agents in the Wild: Safety, Security, and Beyond, the NeurIPS 2026 workshop with the strongest speaker list on agent safety. This piece covers what the workshop is about, maps the places where deployed agents actually break, looks at what the Model Context Protocol (MCP) does and does not protect, and explains why human oversight is still a requirement.
The workshop at a glance
Agents in the Wild runs on Friday, December 11, in Hall 5 of the International Convention Centre in Sydney. It is the third edition, after workshops at ICLR 2026 and ICML 2026 that drew 237 and 327 submissions. Organizers come from UC Santa Cruz, UCLA, UC Berkeley, and Scale AI.
The speaker list is long and senior. It includes Yoshua Bengio (Mila and LawZero), Dawn Song (UC Berkeley), Bo Li (UIUC and Meta Superintelligence Labs), Chen Liang (Google DeepMind), Kai-Wei Chang (UCLA), Qingyun Wu (AG2 and Penn State), Zifan Wang (Meta Superintelligence Labs), and Yunzhong He and Udari Madhushani Sehwag from Scale AI.
The organizers' starting point is that agents are being deployed faster than anyone is studying how to make them safe. The call for papers covers safety and control, security threats including "prompt injection, tool/skill misuse, adversarial manipulation," evaluation, agent tools and protocols, computer-use agents, multi-agent coordination and governance. MCP isn't named, but "tools and protocols" and "tool/skill misuse" put it in scope.
One scheduling note for anyone attending: Who Verifies the Agents?, the other major agent reliability workshop, runs on the same day in the same building.
The problem map: where agents break in the wild
Most agent failures that have reached the public fall into five zones. Each one has a real example, and each one shows a different reason that automated defenses alone haven't been enough.
1. Untrusted content in the context window
In May 2025, Invariant Labs showed that a single malicious issue on a public GitHub repository could hijack an agent connected to the official GitHub MCP server. When the user asked the agent to review open issues, it read the planted instructions, pulled data from the user's private repositories and published it in a pull request on the public one. The leaked details in the demo included private project information and the account owner's salary.
The important part is where the flaw sits. Invariant wrote that "this is not a flaw in the GitHub MCP server code itself," and that "GitHub alone cannot resolve this vulnerability through server-side patches." Every tool did exactly what it was built to do. The agent mixed trusted tools with untrusted text and couldn't tell instructions from data.
2. Poisoned tools
Tool poisoning moves the attack into the tool itself. A malicious MCP server writes instructions into a tool's description, which the model reads but the user rarely sees. The MCPTox benchmark tested this across 45 live MCP servers, 353 real tools and 1,348 malicious test cases on 20 LLM agents. The highest attack success rate was 72.8%, against o1-mini. The best refusal rate of any model was under 3%.
The finding that should worry product teams most: "more capable models are often more susceptible, as the attack exploits their superior instruction-following abilities." Better models follow instructions better, including the wrong ones.
3. Tools that change after you approve them
In September 2025, Koi Security found the first known malicious MCP server in the wild. A package called postmark-mcp, posing as an email integration, worked normally until version 1.0.16, which added one line that blind-copied every outgoing email to an attacker's address. Anyone who had approved the earlier versions got the new behavior automatically.
The same pattern hit developer tools. Check Point Research disclosed MCPoison (CVE-2025-54136) in August 2025. Cursor tied a user's approval to an MCP server's name rather than its contents, so a config could be approved once and swapped for a malicious one later without a new prompt. Cursor's fix in version 1.3 requires re-approval for any change to the config.
4. Over-broad permissions
In the Supabase and Cursor case from July 2025, as summarized by the Cloud Security Alliance, an agent running with a service-role database key read SQL instructions hidden in customer support tickets and leaked integration tokens. The injection was the trigger. The service-role key was what turned it into a breach. An agent that only needed to read tickets had the access to read everything.
5. Goals pursued past scope
The AISI incident is the newest and most unsettling category, because no attacker was involved. Across 122 runs on seven models, 10 runs produced 19 actions outside the test's scope. AISI's disclosure attributes 17 of them to Anthropic's Mythos 5 and two to OpenAI's GPT-5.6 Sol. Besides the open-source attempt, agents messaged real people with malicious payloads and planted prompt-injection content aimed at other AI systems. AISI caught it because general security monitoring flagged data leaving through Tor, and its staff reviewed the activity within minutes.
AISI's conclusion is blunt: "good containment should not depend on the model choosing not to test its boundaries."
Where MCP fits, and where it stops
MCP has become the default way to connect agents to tools, which makes its security model everyone's security model. The specification is clear about intent. On tools, it says that "there SHOULD always be a human in the loop with the ability to deny tool invocations," and that clients "MUST consider tool annotations to be untrusted unless they come from trusted servers." Its security section tells clients to prompt for confirmation on sensitive operations and to "show tool inputs to the user before calling the server."
The gap is between SHOULD and MUST. Human approval is only recommended, and the protocol "does not mandate any specific user interaction model." The Cloud Security Alliance's May 2026 research note lists the structural consequences: no built-in protection against tool poisoning or tools changing silently, no mechanism to force re-approval when a tool definition changes, tool descriptions that reach the model but rarely appear in the interface, and an authorization framework that is optional. It counts at least seven high- or critical-severity CVEs across MCP tooling.
None of that makes MCP uniquely bad. It is a connection protocol, and it is doing what connection protocols do. But it means safety has to come from the system built around the protocol, and in practice that system has a person in it.
Why human oversight is still required
The case for keeping people in the loop rests on four things that have not changed, even as models improved.
Models still can't reliably separate instructions from data. MCPTox's sub-3% refusal rate and the GitHub demo, which used Claude 4 Opus, show that alignment training alone doesn't stop a model from obeying text that looks like an instruction. Invariant's own write-up notes that many prompt-injection detectors missed its attack too.
Automated checkers make the queue smaller, not empty. The data-quality world worked this out first. Scale AI's work on autoraters for expert-level reasoning data is a useful parallel. Using model debate, its system caught 82% of incorrect answers on its hardest dataset, against 23% for a single model, and cut the share of data flagged at final review from 9% to 1%. But debate also flagged correct answers more often, by 29 percentage points on average, and Scale kept at least three human touches on every data point, with humans as the final check. The autorater sorts the work. People still sign off. Agent guardrails are in the same position: they can triage tool calls, and they will be wrong in both directions.
Evaluations can't yet certify an agent as safe. Labelbox's reflections on NeurIPS 2025 named the problems: test data leaking into training sets, models matching patterns instead of generalizing, and benchmarks that don't measure what they claim. The team argued that scores should be treated as "supporting evidence rather than definitive endpoints." An agent that passes a safety benchmark has shown it can pass that benchmark. Behavior in a live environment with real tools is a separate question, which is exactly why AISI's incident happened inside a formal evaluation.
Standards and regulators expect it. The OWASP Top 10 for Agentic Applications, released in December 2025, recommends human approval for high-impact or goal-changing actions and gives a whole category (ASI09) to "Human-Agent Trust Exploitation," where people trust a fluent, confident agent without checking. In the EU, the Digital Omnibus pushed most high-risk AI Act obligations back to December 2027, but human oversight remains one of the requirements those systems will have to meet, and deployers can't hand that duty to a vendor.
Oversight that people click through isn't oversight
There is a catch, and the Agents in the Wild audience knows it well. Human review fails when it turns into a habit.
In the GitHub attack, Claude Desktop asked the user to confirm each tool call by default. Invariant pointed out that "many users already opt for an 'Always Allow' confirmation policy when using agents, and stop monitoring individual actions." The safeguard existed. People had switched it off because it got in the way.
A June 2026 preprint, "Oversight Has a Capacity," models this directly. It argues that "more human oversight can make a system less safe" when a tired reviewer is sent every action, and that attackers can exploit this by flooding the queue with routine requests. The work is a single-author simulation with modeled reviewers, not a human study, but its practical point matches what security teams report: send people fewer, higher-stakes decisions and they make better ones.
The useful question is where in the loop to put the human.
What working oversight looks like
For teams putting agents into production, the incidents above suggest a short set of design rules.
- Gate by blast radius. Require human approval for actions that are irreversible, touch external parties or move data across trust boundaries. Let read-only, reversible actions run.
- Show the actual tool call. The MCP spec already says clients should show inputs before calling a server. A confirmation that says "Allow GitHub?" is not the same as one that shows the repository, the file and the text about to be posted.
- Bind approval to content, not names. Re-approve whenever a server version, tool description or config changes, as Cursor now does.
- Scope permissions per session. Invariant's example policy limits an agent to one repository per session, which would have blocked the GitHub leak.
- Monitor what the agent does, not just what it is asked. AISI caught its incident through outbound traffic, not through the agent's own reports.
- Use automated checks to triage, and people to decide. Measure the reviewers too. If the approval rate is close to 100% and review times are a few seconds, the gate has stopped working.
What to listen for on December 11
If you're attending Agents in the Wild, three threads will tell you the most. First, whether any speaker proposes a way to make instruction and data separation hold at the model level, or whether everyone has accepted that this lives in the system layer. Second, how the Scale AI researchers on the program think about human review at scale, given the company's work on autoraters. Third, governance: Bengio's LawZero has been arguing for AI systems that don't act on their own goals, and that view sits directly against the deployment pace the workshop was set up to examine.
Our view is that the field is heading toward fewer, sharper human checkpoints rather than fewer humans. Agents will take on longer tasks with more tools, and that raises the cost of each unsupervised mistake. The teams that ship agents safely will be the ones that know which ten decisions out of a thousand a person must see, and make sure that person actually looks.
.png)
