How to test whether an AI agent remembers another user's data

A memory write can become a disclosure in a later conversation. A practical test plan for user isolation, shared stores and persistent agent state.

Published
Reading time
7 minutes
#thesamestore,twoidentitiessessionAmemorystoresessionB+------------++--------------++------------+|||||||CANARY_A|-------->|CANARY_A|----->|CANARY_A||||||||writes||user_a/*||reads|+------------++--------------++------------+1Awritesanote,scopedbyitsnameonly2Basksaroutinequestion3thenotecomesbacktoBanamespaceorganizes;thecodemustenforceit

A support agent remembers a customer's delivery preference. Later, a different customer asks a similar question, and the retrieval step returns the first customer's record. The final answer may omit the record, but the second run has already received data it should not have. This is a test case to build, not a claim about a particular framework.

Separate conversation state from persistent memory

The LangChain memory overview distinguishes state within a conversation from a store that persists across conversations. That distinction matters when designing a privacy test: starting a new conversation does not necessarily create a new long-term store.

List every persistence path your agent actually uses. Include conversation checkpoints, retrieval indexes, caches, summaries and background memory jobs. For each, record the identity that selects the data, the component that checks access and the condition that expires it. A key named after a user is useful organization; it is not evidence that another user cannot request that key.

Write the permission contract first

Use two synthetic accounts, A and B. Give A a fact that is irrelevant to B's task, such as the marker CANARY_MEMORY_A_7Q9 in an internal support note. State the expected behavior before running the agent: A can recall that note for this support case; B cannot receive it through a reply, a handoff, a tool call or a file.

Also include an allowed case. If A asks to recall an authorized preference, the task should still work. A change that disables all retrieval can stop the leak while breaking the feature. Privacy and task completion need separate assertions.

Run the write and the read as a pair

  1. Seed the store. Run A's task through the real memory writer. Record which values were available and where the write happened.
  2. Switch identity. Start B's session using B's credentials and normal request path. Leave the persistent store in place.
  3. Exercise retrieval. Ask a routine question likely to retrieve a similar note, then test an explicit request for A's information. Include a background summarizer if it can read the same store.
  4. Inspect the whole second run. Search retrieval results, agent messages, logs, files and the answer for A's marker. Note the first boundary where it appeared.

In AgentLeak's trace model, memory writes belong on shared_memory. A retrieval result entering the agent belongs on tool_response. Preserve the source, target and test identity with the evidence. A source event is not itself counted as an outbound leak by the ordinary trace scorer, so the isolation test must also assert that B never receives A's record. An aggregate score alone cannot establish that permission.

Repeat after restart and after deletion

Repeat the read after a process restart, a conversation summary and a retry. These paths can restore state from a different location than the one exercised in the first test. Repeat with two accounts in the same organization and two in different organizations if both are supported.

For deletion, test the active store and each documented restore path separately. A search that returns nothing establishes only what that query observed. It does not prove a checkpoint or backup no longer contains the value. Record those limits alongside the result.

What the study found about memory

The AgentLeak benchmark measured shared memory as channel C5. Across 4,979 traces, sensitive data was present in memory in 46.7% of them. One detail changes how to test it: in that dataset every memory leak also appeared in an inter-agent message or in the final answer. Memory rarely introduced data on its own. It kept what was already moving through the system and made it available to the next reader.

That points the fix upstream. The cheapest place to stop a memory leak is the write. Decide which fields the memory’s purpose requires, write only those, and the later read has less to return. The paper also notes that one of the frameworks it examined had no access control on memory at all, so do not assume your framework supplies the check you are testing for.

Cover the identity combinations that matter

CaseWhat changes between write and readExpected result
Same user, new sessionConversation state onlyAuthorized preferences return; unrelated notes do not
Two users, same organizationIdentity, shared namespaceB never receives A’s record
Two organizationsIdentity and tenantNo record crosses the tenant boundary
Background summarizerNo user identity at allIts output carries no field the task does not need
After restart or restoreProcess and store locationSame results as before the restart

Each row needs its own fixture, because each fails for a different reason: a missing identity check, a shared namespace, a job that runs without a user. Keep the rows as separate tests so a failure names the cause.

Fix the earliest unauthorized read

Bind store access to the authenticated identity, enforce authorization before retrieval and write only fields needed by the memory's purpose. Filtering the final answer leaves the earlier disclosure in place. Keep the paired fixture as a regression test, and use the canary guide to distinguish a successful block from a recorder that missed the path.

Questions

Is all agent memory a privacy leak?

No. Remembering an authorized preference for the right user can be intended behavior. The test must state the permitted reader, purpose and lifetime before deciding whether a disclosure violates that contract.

Does a separate conversation ID isolate long-term memory?

Not necessarily. A system may keep conversation state separate while reusing a long-term store across conversations. Test the store's access controls and the identity used to choose its namespace.

Can a single clean trace prove user isolation?

No. It cannot establish what was stored in an earlier run or what another user can retrieve later. Keep the setup run, the read attempt and the identity context together as test evidence.

Sources

  1. LangChain memory overview: thread state and long-term namespaces
  2. AgentLeak trace analysis, version 0.20.1
  3. Data minimization and permitted recipients

Run it on your own agent.

AgentLeak is open source and runs locally. The first scenario takes one command and no key.