Instrument once. Test every run.

Install

pip install agentleak
agentleak init
agentleak run --scenario healthcare_patient_summary

# With the local web interface
pip install "agentleak[gui]"
agentleak serve

Use agentleak[gui] when you want the local browser interface. Use the core package for CI, SDK integration or offline trace analysis.

Developer workflow

Configuration reference

Keep the configuration, synthetic traces and policy in the same repository. This makes a score change explainable: reviewers can see whether the agent changed, the detectors changed, or the audited vault changed.

# agentleak.yaml — minimal complete release configuration
project:
  name: support-bot
  description: Privacy regression suite
agent:
  name: support-bot
  type: generic
  endpoint: null
scenarios:
  - id: healthcare_patient_summary
    enabled: true
channels: [user_input, tool_call, tool_response, shared_memory, log, generated_file, inter_agent_message, final_output]
detectors:
  pii: true
  secrets: true
  healthcare: true
  finance: false
  hr: false
detection:
  mode: fast                 # fast | standard | hybrid | llm_only
  presidio: {enabled: false, score_threshold: 0.5}
  llm_judge: {enabled: false, threshold: 0.7}
scoring:
  fail_below: 40
  conditional_below: 70
  block_on_critical: true
  weights: [1, 2, 3, 4]
vault:
  levels: {"1": 40, "2": 12, "3": 5, "4": 2}
  scope_def: customer records reachable by the support workflow
privacy_policy:
  max_risk_index: 0.20
  max_findings: 0
  forbid_levels: [4]
privacy: {redact_values: true, store_raw_traces: false}
reports: {output_dir: reports, formats: [json, html, markdown]}

Validate it with agentleak validate agentleak.yaml. Use the live JSON Schema for editor completion and exact types. Provider keys are resolved from environment variables and should never be serialized into a report.

CLI reference

The CLI is the smallest complete interface for local and CI use. Commands return zero on a passing operation, 1 for a privacy/code-gate failure or operational error, and 2 for invalid usage or a configuration/trace that cannot be resolved.

agentleak init [PATH] [--force]
agentleak validate [CONFIG] [--trace TRACE]
agentleak scenarios
agentleak schema [NAME]
agentleak scan PATH [--mode fast|standard|hybrid] [--format json|sarif] [--fail-under N]
agentleak run [--trace TRACE | --scenario ID] [--config CONFIG] [--format json,html,markdown] [--fail-under N]
agentleak report --input REPORT.json [--format html,markdown]
agentleak history PROJECT [--limit N]
agentleak compare RUN_A RUN_B
agentleak serve [--host HOST] [--port PORT] [--no-browser]

Trace model

Record events at system boundaries. Each event identifies a channel, source, target and content. Preserve ordering and use stable names so leak paths stay comparable across runs.

{
  "run_id": "run_001",
  "agent_name": "support-bot",
  "events": [
    {"channel":"user_input","source":"user","target":"agent",
     "content":"Book a follow-up for Maya Tremblay."},
    {"channel":"tool_response","source":"crm","target":"agent",
     "content":{"email":"[email protected]","sin":"123-456-789"}},
    {"channel":"tool_call","source":"agent","target":"calendar",
     "content":{"email":"[email protected]"}},
    {"channel":"final_output","source":"agent","target":"user",
     "content":"Follow-up scheduled."}
  ]
}

Detection pipeline

Tier 1  deterministic regex + dictionaries + custom rules
Tier 2  canaries, entropy, de-obfuscation and domain recognizers
Tier 2b Presidio recognizers (optional: mode=standard)
Tier 3  LLM-as-Judge semantic detector (optional BYOK: mode=hybrid)

default: fast      = Tier 1 + local deterministic checks
standard           = fast + Presidio
hybrid             = standard + semantic judge
llm_only           = semantic judge only (use only for controlled experiments)

Use fast for every pull request, standard when entity recognition matters, and hybrid only when semantic coverage justifies sending test content to a provider. The judge is not a replacement for deterministic checks and is never enabled by default.

Reports, redaction and data handling

The default is privacy-preserving: findings retain masked values and context, while raw traces are not stored unless explicitly configured. Keep the redaction boundary enabled for hosted runs, use canaries in fixtures, and treat finding metadata as sensitive.

{
  "report": "agentleak",
  "run_id": "run_001",
  "risk_index": 0.44,
  "privacy_score": 56,
  "verdict": "High risk",
  "blocked": true,
  "summary": {"total_findings": 2, "leaked_secrets": 2},
  "findings": [{"channel":"shared_memory","data_type":"diagnosis","level":4,"redacted_value":"dia…sis"}],
  "privacy_policy": {"enabled": true, "passed": false, "violations": []},
  "leak_paths": [{"data_type":"diagnosis","steps":[...] }],
  "remediation_hints": [{"channel":"shared_memory","priority":"critical"}]
}

Use JSON for automation, Markdown for code review, HTML for local investigation and SARIF for source findings. The report schema is available at /api/schemas/analysis-report; the CLI can print every contract with agentleak schema.

Python SDK

from agentleak import AgentLeakRunner, Trace

trace = Trace(run_id="demo", agent_name="support-bot")
trace.add_event(
    channel="tool_call", source="agent", target="crm",
    content={"email": "[email protected]"},
)
trace.add_event(channel="final_output", content="Done")

result = AgentLeakRunner().analyze(trace)
print(result.risk_index, result.privacy_score, result.verdict)

Integrations

The unified agentleak.watch() recorder supports direct channel calls and adapters for major agent runtimes. When an adapter is not available, emit the trace schema directly; AgentLeak does not require a specific orchestration framework.

View adapter examples

BYOK: LLM-judge and OpenRouter

Two independent pieces of AgentLeak can call out to a third-party LLM, and both are bring your own key. Neither is required for the default (regex + entropy + Presidio) pipeline.

# Tier-3 LLM-judge detector (off by default, opt in via --mode)
export OPENAI_API_KEY=sk-...
agentleak run --scenario healthcare_patient_summary --mode hybrid

# Point the judge at OpenRouter or any OpenAI-compatible endpoint instead:
export AGENTLEAK_LLM_BASE_URL=https://openrouter.ai/api/v1
export AGENTLEAK_LLM_MODEL=openai/gpt-4o-mini

# agentleak.yaml — the agent under test, for live (non-scripted) runs
llm:
  provider: openrouter
  base_url: https://openrouter.ai/api/v1
  model: openai/gpt-4o-mini
  api_key_env: OPENROUTER_API_KEY

export OPENROUTER_API_KEY=sk-or-...

CI gate

# agentleak exits non-zero below the threshold
agentleak run --trace traces/latest.json --fail-under 70

# GitHub Actions
- name: Agent privacy gate
  run: agentleak run --trace traces/latest.json --fail-under 70

Keep test traces synthetic and versioned. Compare privacy score, Risk Index, channel findings and leak paths between releases. A regression should fail the build before a leak-prone prompt, tool mapping or memory policy ships.

Cloud API

The hosted service exposes a project dashboard, agent-side endpoints and an integrated API reference. Use the docs page first; use OpenAPI or Swagger when generating clients or validating exact schemas.

Troubleshooting