NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·6 min read
IntentScore: Intent-Conditioned Action Evaluation for Computer-Use Agents
TL;DR
Computer-use agents (AI systems controlling desktop GUIs via language models) make irreversible errors because they don't evaluate action quality before executing. IntentScore, a reward model trained on 398K offline GUI interactions, scores candidate actions by embedding planning intent into action encodings.
✦ Why It Matters
Engineers can use IntentScore to reduce cascading errors in autonomous GUI agents by scoring actions before execution, improving reliability on unseen tasks.
Key Takeaways
- IntentScore embeds planning intent into action encoders to distinguish similar actions with different rationales.
- Train reward models on heterogeneous offline GUI data to generalize across unseen agents and environments.
- Use contrastive alignment and margin ranking as complementary objectives for action quality scoring.
- Deploy as a re-ranker before action execution to prevent irreversible cascading errors in GUI automation.
Full Summary
Computer-use agents leverage large language models to execute graphical user interface operations on desktop environments, but they generate actions without evaluating whether those actions are correct, causing cascading failures. IntentScore addresses this by functioning as a plan-aware reward model—a scoring system that understands both what action is being taken and why (the planning intent).
The model was trained on 398,000 offline GUI interaction steps across three operating systems using two complementary objectives: contrastive alignment (ensuring state-action pairs are relevant) and margin ranking (distinguishing correct from incorrect actions). Architecturally, IntentScore embeds each candidate action's planning rationale into its encoder, allowing it to discriminate between actions that look similar but have different justifications.
On held-out evaluation, IntentScore achieved 97.5% pairwise discrimination accuracy. When deployed as a re-ranker for Agent S3 on OSWorld—an environment completely unseen during training—it improved task success rate by 6.9 percentage points, demonstrating that reward models trained on heterogeneous offline data generalize to new agents and task distributions.