Agentic data science with the Data Agent Kit
cloud.google.com·8h ago
AI Research
arXiv CS.AI·Jul 27·5 min read
From Frame-Level Recognition to Event-Level Confirmation: Repair Traces and Runtime Failure Analysis of Public-Space Gesture Interaction
TL;DR
This research identifies the 'recognition-to-interaction gap' in public-space gesture systems, analyzing failure instances from an interactive kiosk project. It proposes a failure taxonomy and event-level runtime abstraction to improve gesture interaction reliability.
✦ Why It Matters
Evaluate your gesture interaction systems against the identified failure classes to enhance user experience.
Key Takeaways
- Assess gesture systems for the recognition-to-interaction gap.
- Implement the failure taxonomy to identify issues.
- Utilize event-level abstractions for better interaction design.
- Focus on feedback synchronization to improve user experience.
Full Summary
Public-space gesture interaction is often assessed at the frame level, focusing on individual hand movements, but real-world applications require stable interaction events. This study examines eight engineering repair records from a scenic-area interactive kiosk, analyzing four gesture tasks: two-hand bowing, single-hand fist shaking, two-hand catching control, and knowledge-graph node hovering.
From these records, 20 failure instances were identified and categorized into six classes, including model-output degeneration and feedback synchronization failures. The research introduces an event-level runtime abstraction that bridges the gap between hand-landmark recognition models and actual interaction tasks.
While it does not propose a new recognition model or large-scale user evaluations, it provides valuable insights into deployment-grounded failure types and repair mechanisms.