We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·May 4·5 min read
How OpenAI delivers low-latency voice AI at scale
TL;DR
Real-time voice AI requires ultra-low latency and seamless turn-taking, but existing WebRTC (Web Real-Time Communication) infrastructure struggled at scale. OpenAI rebuilt its WebRTC stack—the protocol layer handling audio transmission—to optimize latency, global routing, and conversation flow.
✦ Why It Matters
Engineers building real-time voice systems can learn WebRTC optimization patterns for low-latency, globally-scaled conversational AI.
Key Takeaways
- Optimize WebRTC buffer management and packet prioritization for sub-100ms latency in voice applications.
- Design network adaptation logic to maintain quality across variable bandwidth and geographic regions.
- Implement turn-taking detection to prevent overlapping speech in bidirectional audio streams.
- Test conversational naturalness metrics alongside raw latency when evaluating voice AI infrastructure.
Full Summary
Real-time voice conversational AI requires transmitting audio bidirectionally with minimal delay (latency—the time between speaking and receiving response) while maintaining natural turn-taking, where speakers don't talk over each other. OpenAI's Voice AI product faced a critical gap: standard WebRTC implementations, which handle peer-to-peer audio and video streaming, weren't optimized for the scale, latency requirements, and conversational dynamics needed for production voice agents.
OpenAI rebuilt its WebRTC stack—the underlying protocol layer managing audio transmission—with custom optimizations for buffer management, packet prioritization, and network adaptation. The approach focused on reducing end-to-end latency while maintaining reliability across geographically distributed users.
Results included measurably lower latency metrics and improved turn-taking naturalness, enabling users to interact with voice AI without noticeable delays or awkward pauses. This infrastructure now powers OpenAI's production voice services at global scale.