Benchmark · updated 2026-08-02
Pattern matching calls 49% of real leaks clean.
Run over the 283 scenarios that ship inside the package. Every number below is produced by a script in the repository, with no model in the loop, so re-running it reproduces this page exactly.
Where a leak actually travels
Of the 278 scenarios that leak, 278 leak on internal channels only — the final answer stays clean in every one of them. An audit that reads the output alone reports those as passing.
90.6%tool_call — leaked in 252 of 278 leaking scenarios20.9%shared_memory — leaked in 58 of 278 leaking scenarios20.5%log — leaked in 57 of 278 leaking scenarios4.7%inter_agent_message — leaked in 13 of 278 leaking scenarios4.7%generated_file — leaked in 13 of 278 leaking scenariosThis measures the bundled corpus, not live agents. These scenarios are built to exercise internal channels, so the internal-leak rate is partly by construction — it is the shape of the failure that matters, not the rate as a prediction about your agent.
What the pattern tier misses, per source
Each scenario is scored twice: once against its ground truth, and once with the regex tier alone. The gap is the difference between “no pattern matched” and “nothing leaked” — and it is where a privacy score quietly stops being worth anything.
Built-in examples · 10 scenarios5 scored a perfect 100/100 without ground truth · 5 would not block a gate · 5 fail once it is attachedAgentDojo · 100 scenarios18 scored a perfect 100/100 without ground truth · 64 would not block a gate · 100 fail once it is attachedAgentLeak Bench · 36 scenarios0 scored a perfect 100/100 without ground truth · 13 would not block a gate · 23 fail once it is attachedPII Probes (ai4privacy) · 17 scenarios0 scored a perfect 100/100 without ground truth · 0 would not block a gate · 17 fail once it is attachedPrivacyLens · 120 scenarios90 scored a perfect 100/100 without ground truth · 95 would not block a gate · 119 fail once it is attachedMethod, and what this is not
Every scenario bundled in the package, analyzed by the deterministic pipeline in fast mode (regex tier). Scenarios carrying ground-truth canaries are scored twice: once with them and once without, so the gap between 'no pattern matched' and 'nothing leaked' is measured. No model is called, so re-running this reproduces the numbers exactly.
Not a model rankingNo model is called. This measures detection against fixed traces, so it says nothing about which LLM leaks more. A model-by-model run needs API keys and is a separate exercise.ReproducibleRun python scripts/build_benchmark.py against AgentLeak 0.9.0. Same inputs, same numbers.Deterministic by designScoring uses exact ground-truth matching, not a judge model, which is what makes a regression in CI meaningful rather than noise.