Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·26d ago·5 min read
A Decision-Support Audit Protocol for Supervision Drift in Proxy-Labeled Credit-Risk Prediction
TL;DR
A multi-signal audit protocol was developed to detect supervision drift in proxy-labeled credit-risk prediction models. The protocol effectively identifies performance shifts and calibration issues using a public dataset, enhancing model governance.
✦ Why It Matters
Implement this audit protocol to enhance the reliability of your credit-risk prediction models today.
Key Takeaways
- Adopt a multi-signal audit protocol for credit-risk models.
- Monitor for supervision drift using defined thresholds.
- Utilize calibration diagnostics to improve model accuracy.
- Investigate governance actions based on diagnostic patterns.
Full Summary
Credit-risk models often rely on proxy labels, which can lead to supervision drift—changes in model performance due to shifts in data distribution over time or across segments. This study introduces a locked, multi-signal audit protocol designed to detect and address supervision drift in these models.
The protocol consists of five layers: transfer performance, an oracle-gap probe, calibration diagnostics, feature-label stability, and a synthetic positive control. Using a public LendingClub dataset from 2013 to 2016, the results showed stable rankings and small oracle gaps, with the most significant issue being a mismatch in prevalence and probability scales.
Although the protocol effectively reduced this mismatch through recalibration, the underlying causes remain unclear. The positive control was responsive to larger shifts, but subtler drifts could not be definitively ruled out, indicating a need for further exploration in governance actions.