Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Whisfusion: Parallel ASR Decoding with Masked Diffusion
TL;DR
Autoregressive (AR) models for automatic speech recognition (ASR) face latency issues due to their sequential decoding process. Whisfusion is a new tool that employs masked diffusion techniques to enable non-autoregressive (NAR) ASR decoding, significantly improving speed and accuracy.
✦ Why It Matters
Engineers can leverage Whisfusion for faster and more accurate multilingual ASR applications.
Key Takeaways
- Explore masked diffusion techniques for ASR improvements.
- Consider Whisfusion for multilingual transcription tasks.
- Utilize high-mask specialization in training data.
- Aim for faster processing without sacrificing accuracy.
Full Summary
High-quality multilingual automatic speech recognition (ASR) typically relies on autoregressive (AR) models, which decode speech sequentially, leading to increased latency as transcript length grows. Whisfusion introduces a masked diffusion decoder trained on Whisper-large-v3 audio embeddings, allowing for non-autoregressive (NAR) decoding that bypasses the left-to-right bottleneck.
The model was trained on approximately 68,000 hours of multilingual speech data, focusing on high-mask specialization to align with the fully masked inference process. Results show that Whisfusion surpasses Whisper-large-v3 in average accuracy across English, European, and CJK (Chinese, Japanese, Korean) benchmarks, while running 4-5 times faster.
It also outperforms Whisper-turbo in both accuracy and throughput, achieving performance levels comparable to Canary and Qwen3-ASR, but with 3-7 times faster processing. These findings position masked diffusion as a promising approach for high-throughput multilingual transcription, making it a valuable tool for engineers and researchers in the field.