Testing
Replay a recorded multi-agent run, follow each sensitive value through eight channels — including the ones nobody exports — score it the same way every time, catch copies that were encoded on the way, and attack it on purpose.
Complete trace analysis
AgentLeak replays the whole execution trace and follows every sensitive value through 8 normalized channels. A clean final answer no longer hides a leak in a tool call, shared memory, a log or a generated file.
Traces from any framework normalize to one AgentLeak schema, so the analysis is identical whatever produced the run.
Every finding gets a level from L1 to L4 based on how sensitive the exposed value is and where it went, not a vague red/yellow/green badge.
Each finding carries a remediation hint: prose for your team and a structured, machine-readable hint an agent can apply on its own.
From execution to evidence
Capture disclosures from tool calls, memory and handoffs, then create trace-linked remediation without prompts or manual review.
Ingest frameworks, OpenTelemetry, MCP, logs and CI, then normalize every source into one auditable AgentLeak run.
Analyze a trace from the CLI
agentleak run --trace run.json --output ./reports --format html
open ./reports/run_0001.htmlAgentRisk scoring
AgentRisk is a deterministic, severity-weighted risk index from 0 to 1, defined in a published benchmark. The same trace always yields the same score, so a regression in CI means the agent changed, not the judge.
The score is a closed-form function of the findings and the audited vault. No model decides the number, so it never drifts between runs.
AgentRisk uses the same channels and severity model as the published AgentLeak benchmark, extended with two peer-reviewed datasets, so results are comparable across agents and versions.
Track the score per agent and per release. A rising risk index is an early, quantified warning before anything ships.
AgentRisk over time
Map score thresholds, privacy gates and remediation dependencies across releases so the critical path to a safe deployment stays visible.
Read AgentRisk trends, exposure volume and policy thresholds together to see when a release is approaching unsafe territory.
Score a trace from the CLI
agentleak run --trace run.json --format json --output ./reports
# Risk Index 0.18 · privacy score 82 / 100
# JSON report: ./reports/run_0001.jsonEncoded and obscured leaks
Agents pass each other base64 context blobs, hex identifiers, URL-encoded parameters and values split across messages. A detector that reads only plaintext misses every one. AgentLeak decodes what announces itself, and matches what does not against values the run already exposed.
The same 46 transformed copies (6 data types, 8 transformations) were run against the previous release and this one. A copy counts only when it is reported with the right type and the original value.
Base64, hex, URL-encoding and letter spacing decode to readable text or they do not, so they are decoded and read by the normal detectors. Reversal, ROT13 and splitting look like ordinary text, so they are only matched against values already seen in plaintext earlier in the run.
An encoded copy is attributed to the original value, so it raises exposure on its channel without inventing a second secret. A reversed SIN is the SIN, reversed, not some other SIN.
Measure it yourself
git clone https://github.com/yagobski/agentleak && cd agentleak
pip install agentleak
python scripts/encoded_leaks.py --installed
# agentleak 0.15.0 — 46 of 46 obscured copies detectedAdversarial red-team
Native plugins and Promptfoo privacy/security transpositions map to 46 observable attack classes, then combine with 10 delivery strategies across prompts, tools, RAG, MCP, memory and multi-agent execution — scripted or live.
Select PII, prompt extraction, BOLA/BFLA/RBAC, SQL/shell/SSRF, MCP, memory poisoning or excessive agency, then deliver each probe directly, through guardrail-bypass framing, encoding, Unicode or multi-turn escalation.
Run attacks against a deterministic offline agent for repeatable regression tests, or against a real LLM endpoint to see how your actual agent responds under pressure.
Every probe is analyzed by the same AgentRisk pipeline, then organized into severity, defense rate, delivery method, risk category and a prioritized remediation report.
Start with the deterministic scripted mode: select a vertical, A1, the agent_core preset and the balanced strategy profile, then run ten scenarios. Inspect the stored traces, fix the weakest channel, and repeat the same selection.
Use baseline on every pull request, balanced on protected branches, and complete or evasion on a scheduled security run. Live campaigns require an authorized OpenAI-compatible endpoint and BYOK; scripted campaigns stay offline and deterministic.
Request body for a red-team batch
POST /api/projects/{project_id}/redteam
{
"vertical": "healthcare",
"n": 10,
"adversary_level": "A2",
"plugins": ["pii:direct", "mcp", "agentic:memory-poisoning"],
"strategies": ["basic", "jailbreak-template", "crescendo"],
"mode": "live"
}FAQ