Comparison

They ask whether the agent breaks. We ask where the data went.

promptfoo, Garak and PyRIT are good tools, and all three now reach into agents: they trace tool calls, probe tools and score multi-turn attacks. If you are red-teaming an agent, use one of them. AgentLeak answers a different question. Given a run, did private data cross a boundary it should not have? On which channel, to which recipient, for which purpose? Can you prove it to an auditor, and stop it before the next call?

See the measured gap Try it in 60 seconds

promptfooEvaluation and red-teaming for LLM apps and agents, configured in YAML: 50+ plugins, OpenTelemetry tracing, trajectory assertions on tool use, memory-poisoning and cross-session tests, compliance presets. The broadest adversarial suite here. Use it to find out whether your agent can be pushed into misbehaving.

Attack and grade

GarakNVIDIA's LLM vulnerability scanner: 50+ probe modules with per-probe detectors, and since v0.15 an agent-breaker probe that targets an agent's tools. Use it to scan a model or an agent endpoint for known failure classes.

Probe and detect

PyRITMicrosoft's toolkit for building adversarial campaigns: targets, converters, multi-turn attacks such as Crescendo and TAP, and scorers from LLM judges to classifiers. Use it when you want to compose your own attack campaign, including tool-misuse objectives.

Orchestrate campaigns

AgentLeakPrivacy forensics over a recorded agent run: every event on eight channels, deterministic detectors, a reproducible score, flow rules, and a gateway that enforces the same rules before a tool call is sent. Use it to answer where private data went, on which channel, to whom and for what purpose, with evidence you can attach to an audit.

Audit and enforce

Side by side

promptfooGarakPyRITAgentLeak

Built aroundAttacking an app or agent and grading the outcomeProbing for known failure classesComposing adversarial campaignsAuditing what a run disclosed, event by event

How it sees an agentOpenTelemetry traces; trajectory assertions on tool use and argumentsAgent-breaker probe against the agent's toolsTool-misuse objectives, judged by scorersEvery event on 8 channels: tool calls and responses, inter-agent messages, shared memory, logs, generated files, input, output

What decides a findingRed-team graders (model-based); deterministic assertions availableDetectors attached to each probeScorers: LLM judge, classifiers or customDeterministic detectors and canaries; an LLM judge only if you turn it on

The numberAttack success per pluginProbe hit rateScorer resultsAgentRisk 0–1: severity-weighted, reproducible from the trace

Flow rules (data × sender × recipient × purpose)———Yes, in CI and at runtime

Runtime enforcementMCP proxy that filters and monitors MCP traffic——MCP proxy and OpenAI-compatible LLM proxy: allow, redact, tokenize or block, both directions; tool pinning; signed evidence log

Encoded copies (base64, hex, split…)—Encoding-based attack probesConverters that encode attacksDetected in the run: 46 of 46 on our published benchmark

Regulatory mappingOWASP LLM Top 10, NIST AI RMF, MITRE ATLAS, EU AI Act——14 regulatory frameworks and sector profiles, including GDPR, Québec Law 25, HIPAA, PCI DSS

Adversarial libraryLarge: 50+ pluginsLarge: 50+ probe modulesMulti-turn attacks and converters46 attack classes: smaller

Read the last row honestly. Their adversarial libraries are larger than ours, and if you need breadth of attack, they are the better choice. An em dash means we did not find the capability in their documentation, not that it cannot be done. Checked on 26 September 2026 against each project's own documentation; corrections are welcome as issues on GitHub.

Use both

The natural pipeline runs a model evaluation on the prompts and an AgentLeak gate on the run — and since 0.17 it can be one promptfoo eval: AgentLeak ships a promptfoo python assertion that scores the returned trace. AgentLeak already imports Promptfoo-compatible plugin IDs, so a red-team catalog you have written translates over, and it ingests OpenTelemetry, so a trace your observability stack already captures can be scored without new instrumentation.

Attack layerpromptfoo, Garak or PyRIT: can the agent be pushed into doing what it should not?

Data layerAgentLeak: did private data cross a boundary on any channel, and can you prove it did not?

The gateOne required status check per layer. They answer different questions, so they fail on different bugs.