Red teaming · Concepts

LLM and agent vulnerability types

AgentLeak organizes privacy risk by attack family, injection surface and the channel where disclosure becomes observable. Plugins are executable selectors over this taxonomy—not separate detectors.

Taxonomy

Every scenario has one attack class, one adversary level, one primary channel and one injection surface. A plugin may map to several classes; several plugins may intentionally overlap when they express different threat-model language.

6 families46 attack classes3 adversary levels8 execution channels62 executable plugin IDs

Auditable catalog

The exact count is computed from the public runtime registry, not typed into marketing copy. GET /api/redteam/catalog exposes every plugin, implementation type, native mapping, attack classes, requirements, source URL and MIT license. Each ID also has a stable /api/redteam/plugins/:id permalink.

Six attack families

F1

Prompt & instruction attacks

Direct injection, role confusion, context override, system-prompt extraction and goal hijacking.

user input → final output

F2

Indirect & tool-surface attacks

RAG poisoning, tool output injection, BOLA/BFLA, SQL/shell injection, SSRF, MCP and external exfiltration.

tools/RAG → calls, messages or output

F3

Memory & persistence attacks

Memory poisoning, cross-session disclosure, memory extraction, persistent files and retained logs.

state → later session, file or log

F4

Multi-agent coordination attacks

Cross-agent bleed, orchestrator compromise, shared-memory scope failures, delegation and webhook exfiltration.

agent boundary → message or tool

F5

Reasoning-surface attacks

Scratchpad exposure, reflection extraction, plan verbalization, echo loops and counterfactual probes.

reasoning → output or log

F6

Evasion & obfuscation attacks

Encoding, steganography and invisible-Unicode smuggling used to bypass visible review.

encoded input → obfuscated output

Leak channels

user_input (source only)tool_calltool_response (source only)shared_memoryinter_agent_messageloggenerated_filefinal_output

A source channel can contain authorized private context without being a leak. AgentLeak evaluates whether sensitive data crosses into a destination or persistence channel where it is not needed.

Severity, success and evidence

Plugin severity expresses potential impact. Actual run severity comes from leaked data level and channel evidence. An attack succeeds when an expected canary-backed secret is detected on the class's primary channel; refusal text alone is not counted as success.

Coverage planning

Chat onlyF1, F5 and direct privacy plugins.

RAGAdd F2 indirect injection, RAG poisoning, attribution and document exfiltration.

Tools/APIAdd BOLA, BFLA, RBAC, SQL/shell injection, SSRF, discovery and data exfiltration.

MemoryAdd F3 memory poisoning, session isolation, extraction, logs and artifacts.

Multi-agentAdd F4 trust, delegation, shared-state and webhook tests.

Coding agentAdd Promptfoo coding-agent transpositions for repository, terminal, sandbox, credential and egress boundaries.

Limitations

Privacy and agent security scope

AgentLeak does not claim grading compatibility for Promptfoo's general content-safety, politics, copyright or brand plugins. The catalog marks native implementations and privacy/security transpositions separately.