We’re launching Lyria 3.5 in Google Flow Music, with advances across musicality, lyrics, vocals, and creative control
deepmind.google·6d ago
AI Engineering
OpenAI Blog·Apr 22·5 min read
Speeding up agentic workflows with WebSockets in the Responses API
TL;DR
AI agent loops—systems where models iteratively call APIs to complete tasks—suffered from high latency and redundant API calls. OpenAI implemented WebSockets and connection-scoped caching in the Responses API to reduce overhead by maintaining persistent connections and reusing cached data across requests.
✦ Why It Matters
Engineers building multi-step AI agents can reduce latency and API costs by adopting persistent connections and session-level caching patterns.
Key Takeaways
- Use WebSockets instead of HTTP for agent loops to maintain persistent connections and reduce handshake overhead.
- Implement connection-scoped caching to avoid retransmitting identical context across sequential API calls.
- Profile agent workflows to identify redundant data transfers and API call patterns before optimization.
- Measure latency and API call volume before and after WebSocket migration to quantify gains.
Full Summary
Agent loops are workflows where language models like Codex make repeated API calls to accomplish complex tasks, but traditional HTTP request-response cycles introduce latency and redundant data transfers. OpenAI identified that persistent connection overhead and repeated context transmission were bottlenecks in the Responses API—the interface for structured model outputs.
The team implemented WebSockets, which maintain open bidirectional connections, paired with connection-scoped caching that stores and reuses data within a single session without retransmitting. This approach was applied to the Codex agent loop, a multi-turn reasoning system.
Results demonstrated measurable reductions in end-to-end latency and API call frequency. The optimization is particularly valuable for applications requiring many sequential model invocations, such as code generation, reasoning chains, or autonomous task execution.