Agentic data science with the Data Agent Kit
cloud.google.com·8h ago
AI Research
arXiv CS.AI·Aug 13·5 min read
The Off-Support Barrier: Why Semantic Safety Constraints Are Not Learning-Problem Invariants, and What Follows for Prior Design, Containment, and Verification
TL;DR
A critical examination reveals that semantic safety constraints in AI, such as preventing sandbox escapes, are not invariant under learning problems. This insight leads to new understandings of reward hacking and the limitations of Bayesian prior design.
✦ Why It Matters
Reassess your approach to AI safety by prioritizing hard invariants over soft constraints in model design.
Key Takeaways
- Recognize that semantic safety constraints are not invariant in learning problems.
- Avoid relying solely on Bayesian prior design for enforcing safety constraints.
- Utilize formal verification for local certification of safety constraints.
- Focus on identifying critical off-support regions to improve AI safety.
How It Works
The paper explains that semantic safety constraints are off-support objects, meaning they cannot be measured within the model's data distribution. This leads to challenges in ensuring that AI systems adhere to safety protocols, as traditional optimization methods may inadvertently encourage unsafe behaviors.