Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Liberating LLM Capabilities in Full-Duplex Speech Models
TL;DR
Speech-based large language models (LLMs) are limited to verbal responses, hindering their ability to perform complex tasks. A new approach enables full-duplex speech models to generate structured outputs, allowing for real-time interaction with text-native capabilities.
✦ Why It Matters
Engineers can develop applications that utilize full-duplex speech models for complex, interactive tasks requiring structured outputs.
Key Takeaways
- Explore LWS for applications needing simultaneous speech and text outputs.
- Utilize the Token Schema to enhance existing LLMs without architectural changes.
- Leverage high writing-speaking consistency for better user engagement.
- Test LWS performance on benchmarks like Full-Duplex-Bench and URO-Bench.
How It Works
LWS operates by integrating three channels: listening to audio input, writing text as a primary output, and speaking responses in real-time. This is achieved through a Token Schema that allows the model to process and generate outputs without altering its core architecture.
The training involves a two-stage data pipeline that annotates cognitive processes in sync with the input timeline, enabling the model to maintain context and coherence across modalities.