Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 7·5 min read
The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping
TL;DR
Trace-grounded parametric profiling reveals that video language models, like Gemini 3.6 Flash, struggle with event counting, particularly for transient events. At high counts and frequencies, accuracy drops significantly, highlighting limitations in temporal reasoning.
✦ Why It Matters
Evaluate your video language models using trace-grounded profiling to identify specific temporal reasoning weaknesses.
Key Takeaways
- Implement trace-grounded profiling for precise event evaluation.
- Focus on improving model performance for transient events.
- Analyze high-frequency scenarios to identify accuracy drops.
- Adjust sampling rates cautiously, as they may inflate scores.
How It Works
Trace-grounded parametric profiling involves creating executable event traces for videos, allowing for timestamp-level evaluation of model performance. This method isolates specific event counting tasks, enabling a clearer understanding of where models succeed or fail.