AI agent security testing tools at the end of 2026: what changed, and where privacy fits

promptfoo joined OpenAI, LLM Guard was archived, mcp-scan became Snyk Agent Scan and Presidio changed hands. A sourced review of agent security testing tools in November 2026, and where privacy testing fits.

Published
Updated
Reading time
10 minutes
#threelayers,threequestions+----------------------------------------------------------+|attackcanitbepushedtomisbehave?||||promptfoo,garak,PyRIT|+----------------------------------------------------------++----------------------------------------------------------+|datadidprivatedatacrossaboundary?||||AgentLeak|+----------------------------------------------------------++----------------------------------------------------------+|surfaceiswhatitloadsrisky?||||SnykAgentScan|+----------------------------------------------------------+

The tools for testing LLM agents changed more in 2026 than in the two years before. Several projects changed owners, one was archived, and agent and MCP features arrived everywhere. We checked each project against its own documentation and repository on 14 November 2026. What follows is what we found, with the sources, and our reading of where a privacy layer fits.

What changed this year

  • promptfoo joined OpenAI, announced on 9 March 2026, and states that it remains open source; the repository is still MIT. Its plugins page lists 157 plugins and its strategies page 34 strategies. It now ships an MCP server, promptfoo mcp, with tools such as redteam_run, so an agent testing itself over MCP is no longer unique to any one tool.
  • garak, NVIDIA’s scanner, added agent_breaker in version 0.15.0: a red-team model reads an agent’s tools, generates exploits and checks success through the conversation.
  • PyRIT 1.2.0, released on 13 November 2026, adds an A2A target, so agent-to-agent endpoints can be attacked directly.
  • LLM Guard is archived: the repository is read-only and the README says it is no longer maintained.
  • mcp-scan is now Snyk Agent Scan. The old repository redirects; the scanner needs a SNYK_TOKEN and sends component metadata to Snyk’s API.
  • Presidio moved from Microsoft’s organization to data-privacy-stack/presidio, a community project with a technical steering committee, still MIT.

The gap we looked for

We searched specifically for an open-source tool that captures an agent’s internal channels (tool calls, handoffs, memory, logs, files), scores privacy with a formula anyone can recompute, and gates a pipeline on it. We did not find one. The nearest overlaps are Agentic Radar, which statically scans agent workflows and includes a PII leakage test judged by an oracle model, and DeepTeam, whose PII leakage checks treat the agent as text in and text out with an LLM judge. Commercial runtime products inspect tool and MCP traffic, but do not publish a method or a score.

Attack layer and data layer

LayerQuestionTypical tools
AttackCan the agent be pushed into doing what it should not?promptfoo, garak, PyRIT, DeepTeam
DataDid private data cross a boundary on any channel, and can you show it did not?AgentLeak
Installed surfaceAre the MCP servers and skills an agent loads risky?Snyk Agent Scan

The layers fail on different bugs, so they combine well. AgentLeak ships a promptfoo assertion that scores the trace a promptfoo eval returns, and agentleak import turns the output of promptfoo, Inspect AI, garak or PyRIT into traces it can score.

A practical stack

Most teams do not pick one tool. A reasonable arrangement looks like this.

  1. An attack tool in the evaluation loop. promptfoo, garak or PyRIT, run against the agent on a schedule, to find prompts and tool sequences that push it into misbehaving.
  2. A data check on every run that produces a trace. Feed the attack tool’s output, or your own recorded runs, to a trace-level privacy check, and gate the pipeline on it.
  3. A look at what the agent loads. Scan the MCP servers and skills it connects to before approving them, and again when they change.
  4. A runtime control on the tool path. Whatever the tests find, enforce the rules where the call leaves the process.

The order matters less than the coverage. Each step answers a question the others cannot, and a gap in the stack shows up as a class of bug nobody was looking for.

Where AgentLeak is behind

A comparison is only useful if it says this part too.

  • Breadth of attack. AgentLeak has 46 attack classes and replays scripted turns. It has no attacker model that adapts, where promptfoo and garak do.
  • Leaks in free prose without a model. The deterministic tier finds 237 of 378 planted values in uncued English prose. Tools that grade with a judge handle paraphrase by default; in AgentLeak the judge is optional.
  • Live benchmark environments. Inspect AI runs AgentDojo and others against real agents in sandboxes. AgentLeak’s packs are recorded traces derived from such benchmarks.
  • Scanning what an agent has installed. Snyk Agent Scan inspects MCP servers and skills. AgentLeak’s code scan looks for privacy leaks in source, not for risky tool configurations.

The full matrix, tool by tool, is on the comparison page, with the date and source of every claim.

How we checked, and what we did not

We read each project’s own documentation, release notes and repository on 14 November 2026, and counted what those pages state: the plugin and strategy counts, the licences, the archived status and the ownership changes. We did not install and run every tool against the same agent, so this article makes no claim about detection quality, speed or ease of use. Projects in this field change quickly, and the counts above will move. The comparison page carries the date and source of each claim so that a reader can check it against the live page.

Questions

What is the best open-source tool for AI agent red teaming?

It depends on the question. promptfoo has the widest plugin and strategy catalog, garak is a mature model scanner with an agent probe, and PyRIT is a framework for building attack campaigns. They ask whether an agent can be pushed into misbehaving. A privacy tool such as AgentLeak asks whether data crossed a boundary on any channel.

Is promptfoo still open source after joining OpenAI?

promptfoo states that it remains open source, and its repository was still MIT-licensed when we checked on 14 November 2026.

Can AgentLeak use the output of promptfoo, garak or PyRIT?

Yes. agentleak import turns their output into AgentLeak traces and reports which channels that output could not observe.

Sources

  1. promptfoo red team plugins
  2. promptfoo joining OpenAI
  3. promptfoo MCP server
  4. garak repository
  5. PyRIT repository
  6. Snyk Agent Scan (formerly mcp-scan)
  7. Presidio, now at data-privacy-stack
  8. Agentic Radar
  9. AgentLeak comparison, with every source and date

Run it on your own agent.

AgentLeak is open source and runs locally. The first scenario takes one command and no key.