Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 25·8 min read
The Misattribution Gap: When Memory Poisoning Looks Like Model Failure in Agentic AI Systems
TL;DR
Multi-agent AI systems assume bad behavior stems from model misalignment, but this paper identifies the Misattribution Gap: memory poisoning attacks that masquerade as model failures. Researchers formalized Semantic Norm Drift (SND), where injected policy documents enter shared vector stores and later reappear as trusted context after losing source attribution.
✦ Why It Matters
Engineers must audit memory systems separately from model safety; poisoned knowledge stores evade standard model-focused defenses entirely.
Key Takeaways
- Recognize the Misattribution Gap in AI misconduct analysis.
- Implement Counterfactual Composition Testing for better detection.
- Adopt Memory-Persistent Information-Flow Control to block attacks.
- Understand Semantic Norm Drift as a new threat vector.
How It Works
The Misattribution Gap occurs when memory-layer attacks, which inject misleading documents into AI systems, create behaviors indistinguishable from genuine model failures. Semantic Norm Drift (SND) describes how these documents, once trusted, can lead agents to follow incorrect norms without clear triggers or repeated interactions.
The proposed Counterfactual Composition Testing method effectively identifies the causal entry point of these attacks, while Memory-Persistent Information-Flow Control prevents the misuse of previously trusted information across sessions.