We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Aug 28·5 min read
Introducing gpt-realtime and Realtime API updates
TL;DR
Previously, speech-to-speech AI models had limited real-time interaction capabilities and lacked integration with phone systems or external tools. OpenAI released gpt-realtime, an advanced speech-to-speech model with new API features including MCP server support (Model Context Protocol for tool integration), image input processing, and SIP phone calling support.
✦ Why It Matters
Engineers can now build production voice applications with image understanding, external tool access, and phone system compatibility in a single API.
Key Takeaways
- Integrate external tools and APIs into voice conversations using MCP server protocol.
- Build multimodal voice assistants that process both speech and image inputs simultaneously.
- Connect voice applications directly to phone systems via SIP without separate infrastructure.
- Deploy real-time voice interactions with reduced latency for conversational AI products.
Full Summary
Real-time voice interaction has been constrained by latency, limited input modalities, and poor integration with existing communication infrastructure. OpenAI introduced gpt-realtime, a speech-to-speech model designed for low-latency conversational AI that processes voice input and generates voice output directly.
The Realtime API now supports three major capabilities: MCP server integration (a protocol allowing AI models to call external tools and services), image input (enabling multimodal conversations where users can reference visual content), and SIP phone calling support (Session Initiation Protocol, the standard for VoIP systems). These additions enable developers to build voice assistants that understand context from images, trigger external workflows, and connect seamlessly to traditional phone networks.
The combination addresses fragmentation across voice, vision, and telephony systems that previously required separate integrations.