A measurement instrument for privacy leakage in agent runs.

AgentLeak gives you a fixed trace schema, a bundled corpus with ground-truth values, a score defined by a formula and a benchmark runner that pins all three. A result is a manifest and a number, and someone else can regenerate it byte for byte.

pip install agentleak==0.20.1
agentleak bench init
agentleak bench run bench.yaml

Release 0.20.1, 2026-11-16. PyPI, changelog and source.

RI=11/25=0.44[#][#][][][#][#][][][][][#][#][][][][][#][#][#][][][][][#][#]L4L4L4L4L3L3L2L18secrets,4leaked

The corpus

Every scenario is a complete trace with the values that must not travel. Derived packs keep their source and licence.

PackOriginLicenceScenarios
Built-in scenariosWritten for AgentLeak: five domains, a leaking run and a clean control for eachMIT10
agentdojo_exfilDerived from AgentDojo (Debenedetti et al., NeurIPS 2024 Datasets and Benchmarks)MIT100
agentleak_benchFrom the AgentLeak research benchmark: four verticals, three adversary levelsMIT36
agentsocialbench_maDerived from AgentSocialBench (Wang and Jiang, 2026): private facts handed between agents or into a shared spaceMIT105
injecagent_dsDerived from the data-stealing cases of InjecAgent (Zhan et al., Findings of ACL 2024)MIT96
mcptox_recipient_hijackDerived from MCPTox (Wang et al., AAAI 2026): poisoned tool descriptions that redirect user dataCC BY 4.024
privacylens_ciDerived from PrivacyLens (Shao et al., NeurIPS 2024 Datasets and Benchmarks)CC BY 4.0120
Measured with AgentLeak 0.20.0 on 2026-11-15491

Scenarios that carry ground-truth values are scored twice, with and without them. The gap between the two is the share of leaks a pattern tier does not see, and the benchmark page reports it per pack with its denominator. Dataset credits

Run an experiment

agentleak bench runs the corpus from a manifest, compares two configurations with a paired test, and exports what a paper needs. It adds no dependency; the statistics use the standard library.

StepWhat you getCommand
PinA commented YAML manifest that records the software version and the SHA-256 of every scenario source, with a seed.agentleak bench init
RunRow-level results, a summary, the resolved manifest and a report. Every rate carries its numerator, denominator, a 95% interval and the detector tier. The same manifest and seed give the same rows byte for byte.agentleak bench run bench.yaml
CompareTwo runs paired scenario by scenario: difference with a bootstrap interval, an exact McNemar test, Holm-adjusted p-values and the scenarios whose outcome changed. It warns when the two runs are not comparable.agentleak bench compare runs/a runs/b
ExportLaTeX and Markdown tables, CSV, a Methods paragraph and BibTeX for the software and each pack used.agentleak bench export runs/a
ArchiveA self-contained zip with rows, manifest, environment, checksums and pack licences. bench verify re-runs it and compares hashes.agentleak bench capsule runs/a
Your corpusAgentRisk on a JSONL, JSON or CSV corpus you bring, with the derivation of any Risk Index step by step and its sensitivity to the severity weights.agentleak bench score-corpus corpus.jsonl

Leaking fixtures and clean controls run in the same job. Clean controls come in two kinds, reported with separate denominators: the independent built-in controls, and controls derived from each leaking fixture by removing the events that carry its ground-truth values. The fixture-integrity check is reported on its own and is not presented as detector accuracy. Tutorial with real output

The score

AgentRisk is the ratio of severity-weighted leaked secrets to the severity-weighted vault. Five properties are checked by the test suite.

PropertyStatement
BoundednessRI lies in [0, 1]. It is 0 only when nothing leaks and 1 only when the whole vault leaks.
MonotonicityAdding a leaked secret strictly raises RI.
Severity sensitivityFor the same count, a higher severity gives a higher RI.
Scale invarianceMultiplying every weight by a constant leaves RI unchanged.
Rank robustnessIf one profile dominates another at every level, it scores higher for every positive weight vector.

The report JSON carries the leaked secrets, their levels and the vault, so any RI can be recomputed from it. Weights and level assignments are configurable per deployment and recorded in the report. See the scoring reference and a worked example.

What the detectors find, and what they miss

A score is only as good as the detection under it, so detection quality is published with its misses.

MeasurementWhat it reportsRegenerate
Detection qualityRecall per data type with a 95% interval, in two conditions (structured fields and free prose), plus false positives on independent benign text. Deterministic tier only.python scripts/detection_quality.py
Detection quality, SpanishThe same count on Spanish identifiers, addresses, dates and health terms.python scripts/detection_quality_es.py
Detection quality, FrenchThe same count on a French labelled set.python scripts/detection_quality_fr.py
Cued prose, held outValues written in a sentence after a cue word, in English, French and Spanish. Templates were frozen before the rules were written, and recall is reported on a held-out half.python scripts/detection_quality_cued.py
Credentials and hard negativesProvider token formats found, and how many look-alike strings (UUIDs, hashes, placeholders, version numbers) are wrongly flagged.python scripts/detection_quality_secrets.py
Encoded copiesWhether a known value is still found after base64, hex, reversal, ROT13 or a split across messages.python scripts/encoded_leaks.py
Corpus replayPer-channel leak counts over the bundled corpus, with and without ground truth.python scripts/build_benchmark.py

The scripts live in the source repository, not in the wheel. Clone the tag that matches the installed version. No model is called at the deterministic tier, so a rerun gives the same numbers.

Score your own traces

Bring a recorded run from any framework. The trace schema is public, and adapters cover the common agent libraries.

agentleak validate --trace my_run.json
agentleak run --trace my_run.json --format json
agentleak import results.json --score
agentleak bench score-corpus corpus.jsonl --derive TRACE_ID

A trace is a list of events, each with a channel, a source, a target and content; agentleak schema trace prints its JSON Schema. Declare the vault of sensitive values in the configuration when you know it; otherwise the observed vault is used and the report says so. For paraphrased or inferred leakage, enable the semantic tier and point it at a local model so the data stays in house. Every report names the tiers that ran. Output from promptfoo, Inspect AI, garak or PyRIT goes through agentleak import, which records where each trace came from and names the channels that tool could not observe, so a black-box run is not read as a clean audit of tool calls or memory. See the trace format and the framework adapters.

Limits to state in a paper

These follow from how the corpus and the detectors are built. They belong in a threats-to-validity section.

  1. Labels are defined by construction. A fixture is leaking because its trace was written or derived to contain the planted value on a channel. No human annotated the corpus.
  2. The fixtures are built to exercise internal channels. That 486 of 486 leaking fixtures disclose only internally describes the corpus. It is not a rate to expect from a deployed agent.
  3. The pattern tier is a floor. It recognises formats and field names. A fact that leaks by context, with no recognisable format, is caught only when the scenario carries ground-truth values or when a semantic tier is enabled.
  4. RI depends on the vault. By default the denominator is the set of secrets observed in the trace. Declare the vault explicitly when comparing runs, so the denominator is the same on both sides.
  5. The study’s percentages come from the paper’s own 1,000 scenarios and 4,979 traces. The software corpus is a separate, smaller set. Report which of the two a number comes from.

Cite

Cite the paper for the benchmark and its findings. Cite the software release separately when you report numbers produced with it.

IEEE Access, 2026 · arXiv:2602.11510

AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems

Faouzi El Yagoubi, Godwin Badu-Marfo, Ranwa Al Mallah

@article{elyagoubi2026agentleak,
  title   = {AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems},
  author  = {El Yagoubi, Faouzi and Badu-Marfo, Godwin and Al Mallah, Ranwa},
  journal = {IEEE Access},
  year    = {2026},
  doi     = {10.1109/ACCESS.2026.3704541},
  eprint  = {2602.11510},
  archivePrefix = {arXiv},
  primaryClass = {cs.AI},
  url     = {https://arxiv.org/abs/2602.11510v3},
  note    = {arXiv version 3, revised 15 June 2026}
}