Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 27·8 min read
Position: AI Safety Requires Effective Controllability
TL;DR
Current AI safety focuses on alignment—training models to follow human preferences and policies—but this doesn't guarantee deployed agents can be stopped or overridden in real-world, interactive environments. The paper argues that effective controllability (the ability to constrain or halt a system at runtime) is a distinct and necessary safety requirement beyond alignment.
✦ Why It Matters
Engineers must design explicit runtime control mechanisms alongside alignment to ensure deployed AI agents remain stoppable and overridable in production.
Key Takeaways
- Distinguish alignment (training preferences) from controllability (runtime constraint enforcement) in safety design.
- Build explicit mechanisms to halt or override autonomous agents operating with tool access in production.
- Test safety assumptions in open-ended, interactive scenarios, not just controlled training environments.
- Prioritize runtime authority enforcement as a separate safety requirement from behavioral alignment.
Full Summary
AI safety research has primarily emphasized alignment, the process of training AI systems to follow human values and safety constraints. However, alignment alone does not ensure that a deployed autonomous agent—one operating in open-ended, interactive environments with access to external tools—will actually obey explicit commands to stop, override decisions, or accept new constraints at runtime.
Controllability refers to the technical and operational ability to enforce such runtime authority over an agent's behavior. The position paper identifies a critical gap: a system may exhibit safe behavior in expectation (during training or testing) yet fail to yield to human intervention when deployed in complex, real-world scenarios where it has agency and tool access.
This distinction matters because alignment focuses on what an agent wants to do, while controllability addresses whether humans can force it to stop doing something. The implications suggest that safety frameworks must incorporate explicit mechanisms for runtime constraint enforcement, not just preference alignment during training.