Comparison

They test the model. We test the system.

promptfoo, Garak and PyRIT are good tools, and if you are red-teaming a model you should be using one of them. They share an assumption: the thing under test is a prompt and the answer that comes back. An agent is not that. It calls tools, writes to memory, hands off to other agents and emits logs — and a private value can leave through any of those while the answer stays perfectly clean.

See the measured gap Try it in 60 seconds

promptfooPrompt and model evaluation, plus a red-team suite. Declarative YAML, strong developer ergonomics, large plugin catalog. Use it for evals and adversarial prompts. It grades the answer; it does not model a run as channels.

Model input/output

GarakLLM vulnerability scanner with a wide probe library — jailbreaks, toxicity, leakage of training data. Use it to probe a model. Its unit is a prompt/response pair, not an agent trajectory with tools and memory.

Model behaviour

PyRITMicrosoft's risk identification toolkit: orchestrators, converters and scorers for automated adversarial testing. Use it to build attack campaigns. Scoring is model-graded, so a number moves when the judge does.

Model behaviour, orchestrated

AgentLeakPrivacy forensics over an agent run: eight channels, severity levels, a deterministic score and a CI gate. Use it to answer whether private data left the boundary, on any channel, with evidence you can attach to an audit.

The whole system

Side by side

promptfooGarakPyRITAgentLeak

Unit of analysisPrompt / responsePrompt / responsePrompt / responseA full run, 8 channels

Sees tool calls, memory, logsNoNoNoYes — that is the point

Deterministic scoreAssertion-basedProbe hit rateModel-gradedYes, 0–1 AgentRisk

Ground-truth canariesNoPartialNoYes, exact match

Compliance mappingNoNoNo14 frameworks per finding

Runs with no API keyPartlyPartlyNoYes, fully

Adversarial prompt libraryLargeLargeLarge46 classes — smaller

Model benchmarkingYesYesYesNo, out of scope

Note the last two rows. Their prompt libraries are larger than ours and their model benchmarking is a real feature we do not have. If you need to know which model jailbreaks more easily, use them — not us.

Use both

The natural pipeline runs a model evaluation on the prompts and an AgentLeak gate on the run. AgentLeak already imports Promptfoo-compatible plugin IDs, so a red-team catalog you have written translates over, and it ingests OpenTelemetry, so a trace your observability stack already captures can be scored without new instrumentation.

Model layerpromptfoo, Garak or PyRIT: does the model refuse what it should?

System layerAgentLeak: did private data cross the boundary on any channel, and can you prove it did not?

The gateOne required status check per layer. They answer different questions, so they fail on different bugs.