Accelerating the search for antimicrobial peptides with Google Antigravity
deepmind.google·2h ago
AI Research
arXiv CS.AI·1d ago·5 min read
Can Agents Forget When Asked? A Benchmark for Memory Unlearning in LLM Agents
TL;DR
Bemula, a new benchmark for memory unlearning in large language model (LLM) agents, reveals that current systems struggle to forget requested information effectively. Evaluations show that no system successfully fulfills more than 1.5% of forget requests.
✦ Why It Matters
Evaluate your memory systems against the Bemula benchmark to identify unlearning capabilities.
Key Takeaways
- Assess your LLM agents for memory unlearning capabilities.
- Implement strategies to improve memory deletion accuracy.
- Monitor for potential data extraction vulnerabilities.
- Stay updated on advancements in memory management techniques.
Full Summary
Long-term memory is essential for large language model (LLM) agents, as users often request the deletion of sensitive or outdated information. However, existing benchmarks focus on memory writing and retrieval, neglecting the ability to forget upon request.
To fill this gap, Bemula was developed as the first benchmark specifically for memory unlearning, featuring 50 dialogue instances and 135 forget requests across six types. Evaluations of six mainstream memory systems showed disappointing results, with none passing more than 1.5% of forget requests.
Most systems either left requested information extractable or deleted more than intended. These findings highlight the ongoing challenge of effective memory unlearning in agent design, emphasizing the need for improved memory management strategies.