Research
AgentLeak and AgentRisk are not marketing terms.
The framework and its scoring method come from a published benchmark of privacy leakage across agent execution traces. This tool is the open implementation of that work: the same channels, the same severity model, the same AgentRisk score.
8channels per trace
4severity levels, L1 to L4
283scenarios bundled
3research datasets behind them
3adversary levels, A0 to A2
AgentLeak: measuring privacy leakage across agent execution traces
The primary benchmark. Defines the 8-channel trace model, the L1 to L4 severity levels and the AgentRisk score, evaluated across 36 scenarios in healthcare, finance, legal and corporate domains at adversary levels A0-A2. This tool is the open implementation of that work.
Read on arXivAgentRisk: a deterministic, severity-weighted privacy risk index
The scoring method in full: how findings map to severity, how the vault normalizes the index, and why the score is reproducible. The same trace always yields the same AgentRisk, which is what makes a CI regression meaningful.
See the scoring docsAdversarial channels: prompt injection and exfiltration across a run
The attack families the red-team module replays (prompt injection, tool-response poisoning, memory and hand-off exfiltration) and how each maps to an internal channel and a severity level.
Read the threat model36 scenarios, 4 domains: the benchmark's synthetic data
Every scenario ships with a realistic but fully synthetic vault of PII and PHI, canary values that can only appear if the agent actually leaked them, and adversary levels from a passive A0 to an actively adversarial A2 across healthcare, finance, legal and corporate domains.
See the scenario catalogPrivacyLens: contextual integrity, where the leak is a fact and not a pattern
Shao et al., NeurIPS 2024 Datasets & Benchmarks. An agent pulls private context in through its tools, then acts toward a recipient the norm says must not receive it. 120 of these scenarios ship with AgentLeak, each carrying the dataset's own sensitive_info_items as exact ground truth — because measured on that pack, a pattern-matching tier alone scores most of them a clean 100 out of 100.
See the dataset (CC-BY-4.0)AgentDojo: prompt injection that turns an agent's own tools into the leak path
Debenedetti et al., NeurIPS 2024 Datasets & Benchmarks. A legitimate user task, a planted instruction in data the agent was right to read, and an exfiltration that follows through the agent's own legitimate tools while the user-facing answer stays clean. 100 of these ship with AgentLeak, replayed against the upstream environment and carrying the exact stolen values.
See the dataset (MIT)From severity level to legal obligation: GDPR, Law 25, HIPAA, the OWASP LLM Top 10 and more
Every finding is tied to one of 7 mapped frameworks (GDPR, Quebec Law 25, NIST AI RMF, the OWASP LLM Top 10, the EU AI Act, HIPAA and PCI-DSS v4.0), not a generic red/yellow/green badge, so a compliance review can trace a score straight back to the clause it maps to. This is a mapping to help a review, not a certification.
See the compliance docsAvg RI0.31lower is safer
Privacy68/100 average
Gate1blocked release
Risk Index
Policy gate
Block tool_call and shared_memory exposures above L3.
Active in CICredit where it is due
The datasets behind the bundled scenarios.
Three of the four scenario packs are derived from public research datasets. We ship them reshaped into AgentLeak traces, but the scenarios, the private facts and the attack goals are their authors’ work. Each pack carries its source, licence and attribution in the package itself, and the build scripts that produced them are in the repository so the derivation can be checked line by line.
Cite this work
Referencing AgentLeak or AgentRisk in your own research?
Use the BibTeX entry below for the primary benchmark. The methodology and threat-model write-ups above are companion documents to the same paper, not separate citations.
@misc{agentleak2026,
title = {AgentLeak: measuring privacy leakage across agent execution traces},
author = {AgentLeak},
year = {2026},
eprint = {2602.11510},
url = {https://arxiv.org/abs/2602.11510}
}From paper to practice
Run the benchmark model against your own agent.
The same domains, severity levels and AgentRisk scoring method from the paper are available in the open-source tool. It ships 283 scenarios in total: 10 hand-authored examples (5 with an injected leak, 5 clean controls) plus the full 36-scenario benchmark and two peer-reviewed datasets, all bundled in the package rather than downloaded separately.