Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 6·5 min read
When Shared Rollouts Fail in Defensive Driving Evaluation: A NAVSIM Score Basis Audit
TL;DR
An audit of NAVSIM v2.2 reveals that shared rollouts can lead to misleading defensive driving scores by propagating reference failures into compliance credit. The study emphasizes the need for rigorous score basis and stack disclosure protocols.
✦ Why It Matters
Implement rigorous audit protocols for defensive driving evaluations to ensure accurate performance assessments.
Key Takeaways
- Conduct audits on scoring systems to identify numerical instabilities.
- Implement score basis and stack disclosure for transparency.
- Use blind probes to assess compliance credit accurately.
- Replace unstable solvers to improve rollout reliability.
How It Works
The study examines how shared rollout transformations can lead to compliance credit inflation by propagating failures from human references. By using specific probes, the authors analyze the performance of agents under different conditions, revealing the impact of numerical instability on scoring.