AgentLeak

Benchmark for privacy leakage in multi-agent LLM systems.

This repository accompanies the IEEE Access paper: AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems.

Preprint (v3): https://arxiv.org/abs/2602.11510v3

IEEE Access (2026): https://doi.org/10.1109/ACCESS.2026.3704541

Use AgentLeak on your own agent traces

For the installable privacy-testing tool, see yagobski/agentleak and agentleak.org.

The practical tool provides a Python SDK, CLI, local web UI and CI checks for sensitive data disclosed through tool calls, shared memory, inter-agent messages, logs, generated files and final outputs. Core trace analysis runs locally without an LLM dependency; optional live-agent runs use the endpoint you configure.

Try a bundled synthetic example with Python 3.10+:

pip install agentleak
agentleak run --scenario healthcare_patient_summary

The example illustrates a clean final answer alongside internal-channel disclosures. It is a demonstration, not a production leakage-rate estimate. Follow the tool quickstart to analyze your own traces.

This repository remains the research benchmark and reproduction code. Its experiments, channel definitions and detection pipeline should be cited separately from the practical tool's deterministic analysis and bundled examples.

Paper evaluation scope

The paper (v3) evaluates 1,000 scenarios across healthcare, finance, legal, and corporate domains, with 4,979 validated execution traces across five models. AgentLeak instruments seven channels: C1 final outputs, C2 inter-agent messages, C3 tool inputs, C4 tool outputs, C5 shared memory, C6 logs, and C7 artifacts. The large-scale analysis covers C1/C2/C5; C3/C6 are evaluated on a sample; C4/C7 are examined in an integration study. These channels therefore do not all have the same empirical coverage. The validated trace count is also recorded in paper_stats.json.

Separate unreconciled repository summary (5,694 traces across 5 models)

The table and findings below are preserved from the existing README as a separate repository summary. Their stated 5,694-trace basis and reported rates have not been reconciled with the paper's 4,979 validated traces. Do not interpret them as the paper's validated results or combine the two denominators; consult the paper for its reported results and evaluation scope.

Model	C1 (Output)	C2 (Internal)	H1 (Audit Gap)	Total Leak
Claude-3.5-Sonnet	8.2%	53.9%	45.7%	55.2%
GPT-4o	17.2%	76.8%	59.6%	77.6%
GPT-4o-mini	41.2%	75.3%	34.2%	76.3%
Llama-3.3-70B	26.9%	67.8%	41.3%	89.9%
Mistral-Large	47.5%	96.2%	48.7%	99.3%
Average	28.2%	74.0%	45.9%	79.7%

Findings from the unreconciled repository summary

• Internal channels leak 2.6× more than external (74.0% vs 28.2%)
• Output-only audits miss 45.9% of violations
• Claude 3.5 Sonnet paradox: Lowest C1 leakage (8.2%) but 6.6× internal/external ratio—the highest among all models
• Finding 7 (Tool Leakage): Tool inputs (C3) and system logs (C6) exhibit extremely high leakage rates (up to 85% on Claude 3.5), even when the final agent output (C1) is perfectly sanitized.
• Pattern C2 > C1 holds across all 5 models tested

Scope

• 1,000 scenarios (healthcare, finance, legal, corporate)
• 7 channels: C1 output, C2 inter-agent, C3-C4 tools, C5 memory, C6 logs, C7 artifacts
• 32 attack classes, 6 families
• SDK: CrewAI, LangChain, AutoGPT, MetaGPT

Reproduction

Main Benchmark (C1, C2, C5)

To reproduce the main results (Output, Internal, Memory):

cd benchmarks/ieee_repro
python benchmark.py --n 1000 --traces --model openai/gpt-4o

Advanced Tools & Logs Benchmark (C3, C6)

Targets "Secondary Channel" leakage where sensitive data is sent to external tools or dumped in logs.

cd benchmarks/ieee_repro
# Run for a specific model (e.g., Claude 3.5)
python benchmark_tools.py --n 100 --model anthropic/claude-3.5-sonnet

# Or run the automated multi-model test suite
./run_tools_benchmark.sh

Results are saved in benchmarks/ieee_repro/results/tools/.

Structure

• agentleak/: The core framework SDK
• agentleak_data/: The dataset of 1000 scenarios
• benchmarks/ieee_repro/: Scripts to reproduce the paper's findings, including Finding 7 (Tools & Logs).
• benchmarks/showcase/: Real-world CrewAI integration demo showing the SDK in action.
• paper/: The LaTeX source of the IEEE Access paper

Setup

git clone https://github.com/Privatris/AgentLeak
cd AgentLeak
pip install -e .
pytest tests/ -v

Usage

from agentleak import AgentLeakTester, DetectionMode

tester = AgentLeakTester(mode=DetectionMode.HYBRID)
result = tester.check(
    vault={"ssn": "123-45-6789"},
    output="The SSN is 123-45-6789",
    channel="C1"
)
print(f"Leak: {result.leaked}, Confidence: {result.confidence}")

CLI:

python -m agentleak run --quick --dry-run
python -m agentleak run --full

Reproduction

cd benchmarks/ieee_repro
python benchmark.py --n 100 --traces --model openai/gpt-4o-mini

Traces are in benchmarks/ieee_repro/results/traces/.

Citation

@article{el2026agentleak,
  title   = {AgentLeak: A Benchmark for Internal-Channel Privacy Leakage in Multi-Agent LLM Systems},
  author  = {El Yagoubi, Faouzi and Badu-Marfo, Godwin and Al Mallah, Ranwa},
  journal = {IEEE Access},
  year    = {2026},
  doi     = {10.1109/ACCESS.2026.3704541},
  url     = {https://doi.org/10.1109/ACCESS.2026.3704541}
}

License

MIT