Reimagining service delivery in the agentic era with Google Public Sector
cloud.google.com·21h ago
AI Research
arXiv CS.AI·May 28·8 min read
Data-Efficient On-Policy Distillation for Automatic Speech Recognition
TL;DR
Training large speech recognition models is computationally expensive and data-hungry, limiting practical deployment. Researchers developed an on-policy distillation method that transfers knowledge from a large teacher model to a smaller student model using only relevant data samples.
✦ Why It Matters
Engineers can deploy smaller, faster speech recognition models without sacrificing accuracy by using on-policy distillation techniques.
Key Takeaways
- Explore on-policy distillation for ASR model training.
- Utilize smaller datasets effectively with teacher-student model training.
- Evaluate performance improvements across different language benchmarks.
- Consider the trade-offs between model size and training data requirements.
How It Works
On-policy distillation involves training a smaller student model using the outputs of a larger teacher model during the same training phase. This method allows the student to learn from the teacher's predictions, improving its performance without needing extensive labeled data.