Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·27d ago·5 min read
AdaVSkip: Adaptive Visual Token Skipping Across Layers For Efficient MLLMs Inference
TL;DR
AdaVSkip introduces a novel approach for efficient inference in multimodal large language models (MLLMs) by adaptively skipping visual tokens across transformer layers, achieving a 53.2% reduction in computation while maintaining performance.
✦ Why It Matters
Implement AdaVSkip in your MLLM projects to significantly reduce computational costs without sacrificing performance.
Key Takeaways
- Integrate AdaVSkip to optimize visual token processing in MLLMs.
- Utilize the two-stage training framework for effective router training.
- Combine AdaVSkip with token compression for maximum efficiency.
- Monitor performance metrics to ensure task accuracy is maintained.
Full Summary
Multimodal large language models (MLLMs) face significant computational demands due to the processing of numerous visual tokens across transformer layers. Traditional efficiency methods focus on reducing token counts or employing early exits, but these approaches often overlook the varying redundancy in different inputs and model components.
AdaVSkip addresses this by implementing two lightweight routers in each layer that decide whether visual tokens should pass through or be skipped in self-attention and MLP (multi-layer perceptron) modules. This creates a tailored computation path for each input.
To train these routers effectively, a two-stage framework is employed: the first stage uses supervised learning to establish initial routing policies, while the second stage leverages reinforcement learning to refine these policies based on task performance. The results show that AdaVSkip can reduce FLOPs (floating point operations) by 53.2% on the LLaVA-NeXT-7B model, and when combined with visual token compression, this reduction reaches 91.2%, all while retaining 97.2% of the original model's performance.