NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 28·8 min read
Diagnosing Live Within-Policy Instruction Conflicts in LLM Agents with Witnessed Resolution Profiles
TL;DR
Large language model (LLM) agents often receive conflicting instructions that contradict each other during operation, creating failures that are hard to diagnose. Researchers developed Witnessed Resolution Profiles, a diagnostic method that tracks how agents resolve instruction conflicts in real time.
✦ Why It Matters
Engineers can now systematically diagnose and fix instruction conflicts in LLM agents instead of treating failures as unpredictable.
Key Takeaways
- Implement Witnessed Resolution Profiles to log how your LLM agent resolves contradictory instructions in production.
- Analyze resolution patterns across multiple runs to distinguish intentional policy behavior from bugs.
- Use conflict traces to refine instruction hierarchies and reduce agent failures systematically.
- Track which instructions win conflicts to identify unintended biases in agent decision-making.
Full Summary
LLM agents—systems that use large language models to make decisions and take actions—frequently encounter within-policy instruction conflicts, where multiple directives contradict each other while remaining technically valid under the agent's policy framework. These conflicts degrade agent reliability but are difficult to diagnose because the resolution process happens internally.
Witnessed Resolution Profiles is a diagnostic framework that observes and records how agents actually resolve conflicting instructions during execution, creating a traceable record of decision-making. The method captures which instruction wins, why it was prioritized, and what alternatives were considered.
By analyzing these profiles across multiple runs, engineers can identify systematic patterns in conflict resolution, distinguish between intentional policy behavior and unintended bugs, and refine instruction hierarchies. This approach transforms conflict diagnosis from guesswork into a measurable, reproducible engineering practice.