NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
PANDO: Efficient Multimodal AI Agents via Online Skill Distillation
TL;DR
Multimodal web agents often become inefficient as they accumulate experience, leading to increased computational costs. PANDO, a single-rollout online skill-distillation framework, was developed to address this inefficiency by maintaining a structured Skill Library and employing various techniques like confidence-based skill demotion.
✦ Why It Matters
Engineers can leverage PANDO's techniques to enhance the efficiency of their multimodal AI systems.
Key Takeaways
- Implement structured Skill Libraries for better efficiency.
- Utilize confidence-based skill demotion to optimize performance.
- Adopt cache-aware prompting to reduce computational costs.
- Measure efficiency using new trajectory-level metrics.
Full Summary
Multimodal AI agents, which process and integrate multiple types of data, often face inefficiencies that increase their computational costs during operation. PANDO was created as a solution, utilizing a single-rollout online skill-distillation framework that organizes a Skill Library and incorporates techniques such as progress reflection and cache-aware prompting.
The methodology involved analyzing inefficiencies from VisualWebArena, identifying issues like repeat-action loops and low prompt-cache reuse. In testing, PANDO achieved a 58.3% success rate across 910 tasks, outperforming previous models like SGV and WALT while using 58% and 61% fewer tokens, respectively.
An additional 300-task analysis revealed that structured rules and routines were key to improving success rates, while routing and compression helped reduce token costs. Three new efficiency metrics were introduced to better assess performance beyond just success rates, providing insights into action repetition and prompt utilization.
These findings suggest that optimizing skill management can lead to more efficient AI agents.