Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·21d ago·5 min read
From Task Success to Productive Success: Evaluating Human-AI Collaboration by Quality and Cost
TL;DR
A new framework evaluates human-AI collaboration by comparing outcome quality to interaction costs, revealing significant disparities in productivity. Findings indicate that productive sessions require less user effort and earlier agent probing.
✦ Why It Matters
Evaluate your AI systems using interaction cost metrics to enhance productivity and user experience.
Key Takeaways
- Assess AI productivity by measuring interaction costs alongside outcome quality.
- Recognize that user ratings may not accurately reflect productivity.
- Encourage agents to probe earlier in interactions for better outcomes.
- Tailor interaction strategies based on task-specific quality-cost relationships.
Full Summary
Traditional measures of AI productivity often focus solely on task completion time or outcome quality, neglecting the costs associated with human-AI interaction. This study introduces a productivity-oriented framework that assesses collaboration by evaluating the quality of outcomes relative to the interaction costs incurred.
Analyzing two datasets across four tasks, the research found that sessions with the same quality ratings could have interaction costs differing by up to 70 times. Additionally, the relationship between quality and cost varied by task, with some tasks benefiting from longer interactions while others favored quicker resolutions.
Notably, subjective user ratings were found to be unreliable indicators of productivity. The framework highlights that productive sessions are marked by agents probing earlier and users exerting less effort to correct interactions, thus making interaction costs more visible and informing AI system design.