We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Feb 13·5 min read
Beyond rate limits: scaling access to Codex and Sora
TL;DR
OpenAI's Codex and Sora faced bottlenecks: users hit rate limits (request caps per time window) while the company struggled to track usage fairly across millions of requests. OpenAI built a real-time access system combining rate limits, usage tracking, and a credits model—a prepaid token system—to enable continuous, metered access.
✦ Why It Matters
Engineers can adopt tiered rate limiting plus usage-based credits to scale API access fairly without sacrificing user experience or system stability.
Key Takeaways
- Combine rate limits with usage tracking and credits for fair, scalable API access control.
- Deduct variable credits per request based on compute intensity, not fixed per-request cost.
- Monitor real-time per-user consumption to prevent resource exhaustion and enforce fair quotas.
- Use credits as a prepaid model to enable predictable billing and user-side cost awareness.
Full Summary
Codex (a code-generation AI model) and Sora (a video generation model) experienced demand that exceeded infrastructure capacity. Traditional rate limiting alone—capping requests per minute or hour—created poor user experience and didn't account for varying request complexity or fair resource distribution.
OpenAI designed a three-layer access control system: rate limits enforce hard caps on request frequency; usage tracking monitors actual compute consumed per user in real time; and a credits system (prepaid units deducted per API call) enables predictable billing and prevents runaway costs. The approach treats each API call as consuming variable credits based on output size or compute intensity, not just request count.
This hybrid model allows OpenAI to serve more users equitably, measure consumption accurately, and scale both services without infrastructure collapse. Engineers can now provision access tiers and monitor per-user resource burn in production.