Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
RAS: a Reliability Oriented Metric for Automatic Speech Recognition
TL;DR
Automatic speech recognition (ASR) systems often generate confident but incorrect transcriptions, especially in noisy environments. To address this, a new framework called RAS (Reliability Oriented Metric for ASR) was developed, allowing models to abstain from uncertain transcriptions.
✦ Why It Matters
Engineers can implement RAS to enhance the reliability of ASR systems in challenging environments.
Key Takeaways
- Adopt the RAS metric for evaluating ASR reliability.
- Train ASR models to abstain from uncertain transcriptions.
- Use supervised bootstrapping and reinforcement learning for model training.
- Focus on balancing accuracy with user trust in transcriptions.
How It Works
The RAS metric evaluates transcription reliability by allowing ASR systems to abstain from uncertain segments, thus preventing misleading outputs. This is achieved through a framework that incorporates human preferences to calibrate the trade-off between informativeness and error aversion.