NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Speech Meets ELF: Audio Conditional Continuous-Target Diffusion for Speech Recognition and Translation
TL;DR
Traditional speech-to-text systems generate discrete text tokens, limiting their flexibility. ELF-S2T, a new model based on Embedded Language Flows, processes audio to generate continuous text representations.
✦ Why It Matters
Engineers can leverage ELF-S2T's continuous representation approach to enhance speech recognition and translation systems.
Key Takeaways
- Explore continuous-target models for speech applications.
- Implement audio forcing to improve model training.
- Utilize classifier-free guidance for better inference results.
- Analyze error patterns to refine S2T systems.
Full Summary
Speech-to-text (S2T) systems typically convert spoken language into discrete text tokens, which can restrict their adaptability. To address this, ELF-S2T was developed as an audio-conditioned continuous-target generative model, leveraging the pre-trained Embedded Language Flows (ELF) framework.
It utilizes a frozen Whisper encoder to process audio and a linear projector to integrate audio conditions into the text generation process. During training, audio forcing is applied to reduce reliance on pre-trained text context, while classifier-free guidance enhances audio conditioning during inference.
Experiments on datasets like LibriSpeech and CoVoST2 demonstrate that ELF-S2T performs competitively in both automatic speech recognition (ASR) and speech-to-text translation (S2TT). Notably, error analysis indicates that both ASR and S2TT errors arise from similar issues in the continuous latent space, suggesting a unified semantic mapping process.
These findings could influence future S2T model designs by emphasizing continuous representation generation.