NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
Enhancing Deep Neural Network Reliability with Refinement and Calibration
TL;DR
Deep neural networks often produce overconfident predictions that don't match actual accuracy—a problem called poor calibration. Researchers developed RefCal, a training framework combining a novel loss function with supervised contrastive learning to jointly optimize calibration (confidence alignment), refinement (sharp separation between correct/incorrect predictions), and accuracy.
✦ Why It Matters
Engineers can deploy more trustworthy models by using RefCal to ensure confidence scores genuinely reflect prediction correctness, not just statistical calibration.
Key Takeaways
- Calibration alone doesn't guarantee models understand their mistakes; refinement (sharpness) is equally critical.
- RefCal jointly optimizes three objectives instead of treating calibration separately from accuracy and refinement.
- Use supervised contrastive learning to train the novel loss function promoting refinement explicitly.
- On imbalanced datasets, RefCal achieves 58.81% accuracy with ECE of 0.08, outperforming prior methods significantly.
Full Summary
Neural networks frequently assign high confidence to incorrect predictions, undermining user trust despite strong accuracy metrics. Calibration measures how well predicted confidence aligns with actual correctness probability.
While post-processing techniques can improve calibration scores, they often don't genuinely enhance model understanding. Refinement—the ability to assign distinctly different confidence scores to correct versus incorrect predictions—is a complementary statistical property that calibration methods often degrade.
The authors propose RefCal, a unified training framework that jointly optimizes three objectives: calibration, refinement (sharpness), and accuracy. The method introduces a novel loss function optimized through supervised contrastive learning.
On CIFAR-100-LT with 10% class imbalance, RefCal achieved accuracy of 58.81%, refinement of 95.67%, and expected calibration error (ECE) of 0.08, substantially outperforming the Correctness Ranking Loss baseline (46.27% accuracy, 93.7% refinement, 0.22 ECE).