Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·21d ago·5 min read
VLA-Scope: Shift-Aware Failure Prediction for Vision-Language-Action Models
TL;DR
VLA-Scope is a two-stage framework designed to predict failures in vision-language-action (VLA) models during out-of-distribution (OOD) scenarios. It combines input-shift characterization with execution history, achieving a ROC-AUC of 0.8497 for failure prediction after 60 actions.
✦ Why It Matters
Implement VLA-Scope to improve failure prediction in your VLA models under varying input conditions.
Key Takeaways
- Utilize VLA-Scope for robust failure prediction in VLA models.
- Incorporate execution history to enhance prediction accuracy.
- Evaluate OOD detection performance using ROC-AUC metrics.
- Consider shift classification to better understand input variations.
How It Works
VLA-Scope operates in two stages: first, it detects out-of-distribution (OOD) inputs and classifies their shift categories using pooled representations of images and language. In the second stage, for inputs identified as OOD, it employs a logistic regression model that combines the predicted shift category with features from the action prefix and execution progress, allowing it to dynamically update the risk of failure as the robotic actions are executed.