Data minimization for AI agents: who should see which field

An agent must read private records to do its job. It should not copy them into logs, memory or other agents. How contextual integrity becomes a testable rule, and how it maps to GDPR, HIPAA and Law 25.

Published
Updated
Reading time
9 minutes
#whataschedulerneeds,whatitwassentneededsent------------------------------------------------------------name##appointment##diagnosis.#notneededssn.#notneedednotes..------------------------------------------------------------sentminusneededisthefinding

A privacy rule that says “the agent must not see personal data” is useless for an agent whose job is to handle personal data. The rule that works is older and more precise: data should reach only the actors and surfaces the task requires. Privacy law calls it data minimization. Privacy research calls the underlying idea contextual integrity.

Flows, not secrets

Contextual integrity judges a flow, not a value. The same diagnosis can be appropriate in a message to a treating physician and a breach in a message to a billing tool. So a useful test has to know, for each disclosure, what the data was, where it came from, who received it and for what purpose. That is why AgentLeak records a source and a target on every event, not only the text.

Where data may be, and where it may not go

AgentLeak splits the eight channels of a run into two groups. User input and tool responses form the accessible scope: the agent has to read them to work, so data appearing there is not a leak. The six others are emission surfaces: tool calls, inter-agent messages, shared memory, logs, generated files and the final answer. A finding is a sensitive value that crossed from the first group into the second where no rule allows it.

Without that split, a privacy tool flags every run that reads a record, and teams learn to ignore it. With it, the tool can tell necessary processing from disclosure.

Testing the rule itself

For the software paper, we checked that the scope rule does what it claims. For each of the 261 leaking scenarios in the AgentLeak 0.18.0 corpus, we built a minimized counterpart: the accessible-scope events stay byte for byte identical, and the six emission channels are replaced by acknowledgements carrying no scenario data. None of the 261 counterparts was blocked. When the same accessible bytes were relabeled as logs, 191 of them (73.2%) fired, so the negatives were not empty traces.

This bounds the scope rule on a matched set. It is not a false-positive rate for production text, and we do not present it as one.

Writing the rule down

A flow rule names the data type, the recipient, the purpose, and whether the flow is denied outright. The CI gate and the runtime gateway read the same grammar.

privacy_policy:
  flows:
    - data_type: email
      to: calendar
      for: scheduling
    - data_type: health_condition
      to: "*"
      deny: true

Allow rules are scoped default-deny: once a data type has a rule, any recipient the rules do not list is refused. A data type with no rule at all is recorded and passed, which is why credentials need an explicit deny on servers that should never see them.

How much is too much

Minimization needs a reference for what the task requires. In the benchmark, that reference is the allowed set, rated by two domain experts per field. Across the 1,000 scenarios, the median vault holds 29 sensitive fields and the allowed set holds 3. An agent has to pick about one field in ten from the data it can reach. Any design that hands the whole record to the next component is wrong by roughly a factor of ten before anything malicious happens.

The study also separates what it flags from what a breach is. Sharing more than the task needs is a policy violation of the minimization principle, even when no outsider sees the data. It widens the attack surface, which is a risk. Only data reaching an unauthorized party is a confidentiality breach. A report should keep the three apart, because the response to each differs.

Strict is not the same as minimal

A rule that forbids any sensitive data from moving between agents is simple and breaks the system. In the paper’s tradeoff scenarios, a strict setting that prevented any sensitive sharing between agents pushed task success down to between 35% and 65%. A selective rule did far better. A prototype that masked fields outside the allowed set on internal channels cut internal leakage from 31.5% to 2.4% for a 4.7 point drop in task success. The lesson is to specify what each recipient needs, not to starve all of them.

Practically, write the allowed set with the person who owns the workflow, not only with the engineers. They know that the billing agent needs an insurance identifier and the scheduler does not. That knowledge is the policy.

How it maps to the law

  • GDPR. Article 5(1)(c) requires data to be limited to what is necessary, and Article 25 requires data protection by design and by default.
  • HIPAA. The minimum necessary standard limits uses and disclosures of protected health information to what the purpose requires.
  • Québec Law 25. The amended private-sector act requires privacy by default and limits collection and communication to what is necessary.

AgentLeak maps each finding to the obligations it touches across 14 frameworks and sector profiles. That gives a reviewer a dated record of which flows were tested and which failed. It does not replace the review.

Questions

What is contextual integrity?

A privacy theory from Helen Nissenbaum: information flows carry norms tied to the context they come from. A diagnosis shared with a physician is appropriate; the same diagnosis shared with an employer is not. Privacy is breached when a flow violates the norms of its context.

Does data minimization mean an agent cannot access personal data?

No. An agent that schedules a medical appointment needs the patient’s details. Minimization means each recipient, whether a tool, another agent, a log or a memory store, receives only the fields its part of the task requires.

Does passing an AgentLeak policy make an agent GDPR compliant?

No. The mappings identify which obligations a finding touches, to support an engineering or governance review. Compliance is a legal assessment that needs human review.

Sources

  1. Nissenbaum, Privacy as Contextual Integrity, Washington Law Review, 2004
  2. Contextual integrity rules in AgentLeak
  3. Privacy and compliance documentation
  4. AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems, IEEE Access, 2026 (arXiv v3)

Run it on your own agent.

AgentLeak is open source and runs locally. The first scenario takes one command and no key.