Privacy · Compliance engineering

Privacy compliance with trace-linked evidence

AgentLeak evaluates what an agent actually did across prompts, tools, memory, messages, logs and files, then links each observed disclosure to deterministic policy assertions and regulatory controls. The result is an engineering evidence package—not a legal certification.

Safe interpretation

observed_clear means no configured control was triggered in the tested trace. It never means the organization, model or all future behavior is legally compliant.

Why this is different from a generic red-team grader

General red-team platforms are excellent at generating broad malicious prompts. Privacy compliance requires additional evidence: where data entered, which execution boundary it crossed, whether that boundary was allowed, which stable finding proves the event, and which control needs review.

Full traceEight normalized channels cover tool arguments and responses, shared memory, inter-agent messages, logs, generated files and final output.

Deterministic joinsEvery mapped control links to stable finding IDs instead of relying only on a free-form grader explanation.

Governance gapsUnconfigured purpose and vault assertions are marked not_assessed rather than silently treated as tested.

Local-firstRegex, canary, entropy, policy and compliance evaluation stay local by default; semantic judging is explicit BYOK.

ReproducibilityThe evidence manifest carries a canonical SHA-256 digest for artifact comparison without claiming a signature.

Assurance model

trace_only

Leak detectors and channel evidence ran, but governance assertions were not configured.

trace_and_policy

The trace was evaluated together with explicit privacy assertions such as forbidden channels, data types and audited vault scope.

not_assessed

A control needs configuration that was absent. This is a visible evidence gap, not a pass or a failure.

The legacy per-framework compliant/non_compliant field remains for CI compatibility. Use compliance.assurance when presenting the strength and scope of the evidence.

Finding-to-control evidence matrix

Each at-risk control contains redaction-safe evidence_details: finding IDs, channels, data types, levels and policy rules. The top-level matrix provides the inverse index—one finding to every affected framework and control.

"compliance": {
  "assurance": {
    "status": "controls_at_risk",
    "evidence_grade": "trace_and_policy",
    "controls_not_assessed": 0
  },
  "evidence_matrix": [{
    "finding_id": "fnd_7ac1",
    "frameworks": ["gdpr", "law25"],
    "controls": ["gdpr.art5.1b", "gdpr.art5.1f", "law25.confidentiality"]
  }],
  "integrity": {
    "algorithm": "sha256",
    "digest": "…",
    "signed": false
  }
}

Integrity, not attestation

The digest detects accidental artifact drift when recomputed over the canonical fields. Because it is unsigned and stored beside the report, it is not tamper-proof and does not establish third-party provenance.

Turn privacy obligations into deterministic assertions

Configure only boundaries the system owner can state truthfully. AgentLeak currently maps forbidden channel/data-type violations to GDPR purpose limitation and explicit vault requirements to privacy by design.

# agentleak.yaml — deterministic assertions evaluated after every run
privacy_policy:
  max_risk_index: 0.20
  max_findings: 0
  forbid_levels: [4]
  forbid_channels: [log, shared_memory]
  forbid_data_types: [llm_api_key, credit_card]
  require_explicit_vault: true

forbid_channelsData must not persist in logs, shared memory or generated files.

forbid_data_typesSelected categories may not leave the authorized source boundary.

forbid_levelsCritical or special-category data is release-blocking.

require_explicit_vaultRisk scoring must use an audited reachable-data denominator, not the observed fallback.

max_risk_indexThe weighted disclosure density must remain below the release threshold.

Framework and sector coverage

The same observed findings are mapped to GDPR, Québec Law 25, NIST AI RMF, OWASP LLM Top 10, EU AI Act, HIPAA, PCI-DSS, FERPA, COPPA, GLBA, TCPA, plus insurance, telecom/CPNI and real-estate privacy profiles. Controls are transparent predicates over leaked level, data type, channel, Risk Index and policy violations; no hidden compliance grader decides the result.

One event, several obligations

A health identifier written to shared memory can affect minimisation, confidentiality, special-category processing, HIPAA minimum-necessary and security controls. The matrix keeps the single finding as the source of truth while showing every mapped obligation.

DPO and engineering workflow

1Scope

Declare purpose, reachable vault, prohibited channels/data types and authorized test target.

2Exercise

Run baseline scenarios plus red-team plugins matching tools, RAG, memory, roles and data access.

3Review

Start from at-risk controls, open linked finding IDs and reconstruct the leak path.

4Remediate

Minimize tool schemas, isolate memory, redact persistence channels and enforce authorization.

5Regress

Repeat the same vault, plugins, strategies and target configuration; compare scores and evidence.

6Retain

Export redacted JSON/HTML/Markdown artifacts under the organization’s evidence-retention policy.

Enforce the privacy contract in CI

Fail on deterministic assertions or selected framework mappings. Keep the JSON report as the machine artifact and publish HTML/Markdown only after verifying redaction settings.

# Assertions block the run
agentleak run --trace traces/latest.json --config agentleak.yaml --fail-under 80

# Inspect evidence gaps and the manifest
jq '.compliance | {assurance, evidence_matrix, integrity}' reports/*.json

What AgentLeak does not prove

Legal statusNo report is legal advice, certification, attestation or proof of organization-wide compliance.
Untested pathsA passing trace says nothing about workflows, tenants, roles, languages or attack paths that were not exercised.
Detector recallUnknown, encrypted or semantic data may require canaries, Presidio, custom detectors or an explicitly approved LLM judge.
ProvenanceThe built-in digest is reproducible but unsigned. External signing and controlled evidence storage remain deployment responsibilities.
Regulatory scopeConfirm applicable laws, lawful basis, retention, DPIA and data-subject obligations with qualified privacy counsel.