Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 21·6 min read
OEP: Poisoning Self-Evolving LLM Agents via Locally Correct but Non-Transferable Experiences
TL;DR
Memory-augmented LLM agents (language models with stored experience logs) use self-reflection to improve over time, but existing poison attacks require detectable malicious content or system access. Researchers discovered OEP (Observation Exploitation Poisoning), which injects experiences that appear locally correct—solving immediate tasks well—but fail to generalize to new problems, degrading agent performance without triggering safety filters.
✦ Why It Matters
Engineers deploying self-improving LLM agents need to validate that stored experiences generalize beyond immediate contexts, not just appear locally correct.
Key Takeaways
- Validate agent experiences for generalization, not just local correctness, before storing in memory.
- Monitor agent performance degradation on novel tasks as indicator of memory poisoning.
- Implement out-of-distribution detection for stored experiences to catch non-transferable patterns.
- Design safety filters to catch subtle semantic failures, not just explicit malicious content.
Full Summary
Memory-augmented large language model agents—systems that store past interactions and learn from them—use iterative reflection to solve complex tasks. However, this self-evolution mechanism creates security gaps.
Prior attacks on agentic memory require either privileged system access or obviously malicious content, both detectable by modern safety filters. This work identifies an underexplored attack surface: inducing agents to generate experiences that appear correct in isolation but fail to transfer to new problems.
The OEP technique exploits this by poisoning the agent's memory with locally valid but non-generalizable experiences. These experiences pass local validation checks yet degrade performance when the agent encounters novel tasks, effectively corrupting the agent's learned knowledge without triggering content-based defenses.