Software replay · AgentLeak 0.16.0 · 2026-09-27
Pattern matching calls 31% of real leaks clean.
Run over the 266 scenarios bundled with AgentLeak 0.16.0. Every number below is produced by a script in the repository, with no model in the loop, so re-running that release and configuration reproduces this snapshot.
The published paper evaluates 1,000 research scenarios and 4,979 model traces; read those results separately. Latest verified software release: 0.17.0.
Where a leak actually travels
Of the 261 scenarios that leak, 261 leak on internal channels only — the final answer stays clean in every one of them. An audit that reads the output alone reports those as passing.
98.5%tool_call — leaked in 257 of 261 leaking scenarios15.7%shared_memory — leaked in 41 of 261 leaking scenarios15.3%log — leaked in 40 of 261 leaking scenarios5%inter_agent_message — leaked in 13 of 261 leaking scenarios5%generated_file — leaked in 13 of 261 leaking scenariosThis measures the bundled corpus, not live agents. These scenarios are built to exercise internal channels, so the internal-leak rate is partly by construction — it is the shape of the failure that matters, not the rate as a prediction about your agent.
What the pattern tier misses, per source
Each scenario is scored twice: once against its ground truth, and once with the regex tier alone. The gap between "no pattern matched" and "nothing leaked" is where a privacy score quietly stops being worth anything.
Built-in examples · 10 scenarios5 scored a perfect 100/100 without ground truth · 5 would not block a gate · 5 fail once it is attachedAgentDojo · 100 scenarios7 scored a perfect 100/100 without ground truth · 45 would not block a gate · 100 fail once it is attachedAgentLeak Bench · 36 scenarios0 scored a perfect 100/100 without ground truth · 13 would not block a gate · 23 fail once it is attachedPrivacyLens · 120 scenarios62 scored a perfect 100/100 without ground truth · 67 would not block a gate · 119 fail once it is attachedMethod, and what this is not
Every scenario bundled in the package, analyzed by the deterministic pipeline in fast mode (regex tier). Scenarios carrying ground-truth canaries are scored twice: once with them and once without, so the gap between 'no pattern matched' and 'nothing leaked' is measured. No model is called, so re-running this reproduces the numbers exactly.
Not a model rankingNo model is called. This measures detection against fixed traces, so it says nothing about which LLM leaks more. A model-by-model run needs API keys and is a separate exercise.ReproducibleRun python scripts/build_benchmark.py against AgentLeak 0.16.0. Same inputs, same numbers.Deterministic by designScoring uses exact ground-truth matching, not a judge model, which is what makes a regression in CI meaningful rather than noise.