We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Mar 23·5 min read
Creating with Sora Safely
TL;DR
Video generation models like Sora create novel safety risks—deepfakes, misinformation, and misuse on social platforms—that existing safeguards don't address. OpenAI built Sora 2 and the Sora app with safety embedded from the start, including content filters, usage policies, and detection systems.
✦ Why It Matters
Engineers building generative video systems can adopt safety-first design patterns combining detection, policy enforcement, and user controls.
Key Takeaways
- Embed safety constraints in model architecture and app design, not as post-hoc filters.
- Implement layered protections: policy enforcement, automated detection, authentication, and rate-limiting.
- Test defenses against known attack vectors and real-world harm scenarios before public release.
- Monitor usage patterns and user behavior to detect and respond to emerging misuse.
Full Summary
Sora, a state-of-the-art video generation model (AI that creates video from text descriptions), introduces safety challenges absent in prior systems: synthetic video can enable deepfakes, spread misinformation, and facilitate abuse at scale on social platforms. OpenAI developed Sora 2 and a companion Sora app with safety as a foundational design principle rather than an afterthought.
The approach combines multiple concrete protections: content policy enforcement, automated detection of policy violations, user authentication and rate-limiting to prevent abuse, and design choices that discourage misuse. The team anchored decisions in real-world harms and tested protections against known attack vectors.
While specific metrics on reduction of harmful outputs remain undisclosed, the layered defense strategy aims to balance creative capability with risk mitigation for a public-facing platform.