Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 28·8 min read
Intelligence as Managed Autonomy: Failure, Escalation, and Governance for Agentic AI Systems
TL;DR
Autonomous AI agents lack clear failure modes and governance structures, creating risks when systems operate without human oversight. This work proposes a managed autonomy framework that treats intelligence as a spectrum requiring escalation protocols—mechanisms to route decisions back to humans when confidence drops or stakes rise.
✦ Why It Matters
Engineers can design safer autonomous systems by implementing escalation protocols and confidence thresholds rather than choosing between full autonomy or manual control.
Key Takeaways
- Implement escalation protocols that route decisions to humans when agent confidence drops below thresholds.
- Design failure detection mechanisms to identify when autonomous systems exceed their competence boundaries.
- Establish governance structures defining which task types require human approval before execution.
- Deploy autonomous agents incrementally using graduated autonomy rather than binary autonomous-or-manual modes.
Full Summary
Agentic AI systems—software that independently plans and executes multi-step tasks—currently lack standardized approaches to handling failures and maintaining human control. Existing systems often operate in binary modes: fully autonomous or fully manual, with no middle ground.
This work introduces managed autonomy, a governance model treating intelligence as a graduated capability rather than an on-off switch. The framework incorporates escalation protocols that automatically route decisions to humans when confidence thresholds drop, task complexity exceeds predefined limits, or potential harms increase.
Key components include failure detection mechanisms to identify when agents exceed their competence boundaries, and governance structures defining which decisions require human approval. The approach enables organizations to deploy autonomous systems incrementally while maintaining oversight, reducing both safety risks and operational bottlenecks from unnecessary human intervention.