Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·27d ago·5 min read
Modeling, Scaling, and Decoding: Optimizing Controllable Speech Generation with Nonverbal Vocalizations
TL;DR
An NVV-aware DiTAR system was developed to enhance controllable speech generation by effectively modeling nonverbal vocalizations (NVVs). It achieved top rankings in the ISCSLP 2026 NVVSpeech Challenge, demonstrating significant improvements in NVV synthesis accuracy.
✦ Why It Matters
Consider implementing NVV-aware models to enhance the expressiveness of your speech synthesis applications.
Key Takeaways
- Explore NVV-aware models for improved speech synthesis.
- Utilize targeted synthetic data augmentation for underrepresented categories.
- Balance NVV correctness and perceptual quality in candidate selection.
- Implement frequency-aware rebalancing in training datasets.
How It Works
The NVV-aware DiTAR system models continuous speech latents and encodes NVV categories as tokens, allowing for precise control over vocalizations. It employs a two-step training process: first, large-scale bilingual pre-training on diverse NVV speech, followed by fine-tuning with synthetic data to enhance representation.
During inference, it optimizes the generation process by selecting acoustic prompts and adjusting parameters to minimize errors.