How the AgentLeak benchmark was built: 1,000 scenarios, 4,979 traces and what we checked
How the scenarios, the vault and the allowed set were designed, how three detection tiers were calibrated and validated, and where the benchmark’s own limits are.
A benchmark is a measuring instrument, and an instrument deserves the same questions as any other: what does it measure, how was it calibrated, and how wrong can it be. This article walks through how the AgentLeak benchmark was built, using the paper’s own account, because the headline rates only mean something if the construction holds.
What a scenario contains
A scenario is a structured record with seven parts. A task description in natural language, written to sound like a production request: schedule a follow-up and coordinate with billing. A private vault of every sensitive field, each with a category label such as protected health information or payment card data, and a sensitivity weight. An allowed set of the minimal fields the task legitimately needs. Tool specifications with deterministic mock behavior. An agent topology. An attack level, from benign to adversarial. And ground-truth labels of which channels should leak under which defenses.
The vault and the allowed set are what make this a data minimization test. Across the 1,000 scenarios, the median vault holds 29 fields, ranging from 21 to 39, and the allowed set holds 3, ranging from 2 to 5. That is a median ratio of 9.7 to 1. An agent has to pick a small subset out of a large pool of sensitive data, which is the situation where over-sharing happens.
Four domains, 250 scenarios each
Healthcare covers appointment scheduling, insurance claims, care coordination and laboratory results. Finance covers know-your-customer and anti-money laundering workflows, disputes, underwriting and regulatory reporting. Legal covers contract analysis, discovery, client onboarding and privilege protection. Corporate covers human resources records and similar internal work. The verticals were chosen for their stringent privacy requirements, and the compliance frameworks that apply differ: HIPAA and Québec Law 25 for health, GLBA and PCI-DSS for finance.
Synthetic data that behaves like real data
Values come from the Faker library with domain extensions: social security numbers in the three-two-four format with valid area numbers, card numbers that satisfy the Luhn checksum, medical record numbers matching hospital conventions. To check that detection works on realistic distributions, the authors also drew on two public datasets of pre-anonymized records, one of 3,624 personal data samples and one of 2,202 clinical notes.
Sensitive values come in three tiers, because a detector has to work across all of them. Explicit canaries are unique markers, trivially detectable if they leak verbatim, and validate the capture infrastructure. Format-valid fakes are syntactically correct identifiers that test format-based detection. Natural-language facts are plausible statements that need semantic understanding, and test the full pipeline.
Who decides what is allowed
The allowed set is a judgment, so it was made by people. Two independent domain experts per vertical, a healthcare compliance officer, a financial auditor, legal counsel or a corporate privacy officer, rated each vault field on a four-point scale from definitely not required to definitely required. A field entered the allowed set only if both rated it three or higher. Agreement between raters, measured with Cohen’s kappa, was 0.82 for healthcare, 0.79 for finance, 0.81 for legal and 0.75 for corporate, which the paper describes as substantial agreement.
A detector with three tiers
No single method catches both a verbatim copy and a paraphrase, so the pipeline has three tiers, applied to all seven channels. Tier 1 matches the canaries with a regular expression: high confidence, no false positives, and a check that the recorder worked. Tier 2 extracts known sensitive formats such as social security numbers, Luhn-valid card numbers and ICD-10 codes. Tier 3 asks an LLM to judge whether a passage discloses a vault field even when paraphrased.
A leak is attributed to the lowest-numbered tier that catches it, so each violation is counted once. The reported distribution, 8% at Tier 1, 10% at Tier 2 and 82% at Tier 3, is therefore the marginal contribution of each tier after the cheaper ones have taken what they could. It also says something plain: most leaks in agent traces are semantic, not formatted.
How the judge was calibrated and checked
- Threshold. The semantic similarity threshold was tuned by grid search on 200 trace segments labeled by a domain expert, 100 positive and 100 negative, taken from pilot runs and held out entirely from the 1,000-scenario evaluation.
- Test set. On a separate set of 1,000 segments, Tier 3 reached a false positive rate of 4.8% and a false negative rate of 7.4%.
- Other judges. Inter-judge agreement was checked across Qwen-2.5-7B, Llama-3.1-8B and GPT-4o-mini, and temperature variation between 0.0 and 1.0 moved leakage rates by less than 3%.
- Human review of results. A domain expert reviewed a random sample of 100 detected leaks, 50 internal and 50 external, and confirmed 94% as genuine. The six disagreements were edge cases around paraphrased medical terminology.
The sensitivity table in the paper is worth reading for what it shows about the trade. Raising the threshold lowers false positives and raises false negatives, and the chosen setting is the highest value that keeps false positives under 5% while the F1 score stays near its maximum. The detection tiers article gives the measured numbers for each configuration.
Running it
The evaluation ran between October 2025 and January 2026. Each scenario ran on all five models: GPT-4o, GPT-4o-mini, Claude 3.5 Sonnet, Llama 3.3 70B and Mistral Large. After 21 traces were excluded for timeouts or incomplete capture, 4,979 validated traces remained, evenly spread: about 1,245 per vertical and between 995 and 996 per model. Each trace holds both a single-agent execution and a multi-agent one of the same scenario on the same model, which is what allows a paired comparison between architectures.
Where the benchmark is limited
The authors list their own limits, and they bear on how to read every rate in the paper.
- English only. The judge threshold was calibrated on English scenarios and might need retuning elsewhere, and named-entity coverage for non-English identifiers has gaps.
- One topology. Every scenario uses a coordinator and a worker. Hierarchical and peer-to-peer systems may differ.
- An LLM judging LLMs. Because 82% of detections are semantic, the pipeline inherits known judge biases. The deterministic tiers, the held-out calibration and the human review are there to bound that, not to remove it.
- Conservative by construction. With a 7.4% false negative rate and a small judge, the reported leakage rates are lower bounds.
None of this lowers the finding that internal channels leak. It tells you which direction the error runs. If anything, the true rates are higher.
Questions
How many traces were in the final analysis?
4,979. The 1,000 scenarios ran on five models, which is 5,000 model and scenario pairs, and 21 traces were excluded because of timeouts or incomplete capture. No systematic pattern by domain or attack level was seen among them.
Why use a small judge model?
Cost. Qwen-2.5-7B was the default judge, and the authors checked that it did not introduce systematic bias by validating with Llama-3.1-70B-Instruct and GPT-4o-mini. GPT-4o-mini reached a lower false negative rate, so the reported leakage is a conservative estimate.
Is the benchmark data real?
It is synthetic, built with the Faker library and domain extensions so identifiers follow valid formats, and checked against real distributions from public datasets of pre-anonymized records. No real person’s data is in a scenario.
Can I reproduce the results?
The scenarios carry cryptographic hashes, seeds and versioned prompts, and the research repository holds the materials. Reproducing the live-model numbers needs access to the same models.
Sources
Run it on your own agent.
AgentLeak is open source and runs locally. The first scenario takes one command and no key.