Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 3·5 min read
Shiva-DiT: Residual-Based Differentiable Top-$k$ Selection for Efficient Diffusion Transformers
TL;DR
Shiva-DiT introduces a novel approach for Diffusion Transformers that utilizes Residual-Based Differentiable Top-k Selection, significantly improving efficiency. It achieves a 1.54x speedup while maintaining competitive fidelity across various datasets.
✦ Why It Matters
Consider implementing Shiva-DiT to enhance the efficiency of your Diffusion Transformer models today.
Key Takeaways
- Explore the implementation of Shiva-DiT for your models.
- Utilize the Context-Aware Router for dynamic token retention.
- Leverage the Adaptive Ratio Policy for optimized performance.
- Achieve significant speedups while maintaining model fidelity.
How It Works
Shiva-DiT employs a Residual-Based Differentiable Top-k Selection method that allows for efficient token pruning during the forward pass. This mechanism uses a straight-through estimator to propagate gradients effectively, ensuring that both token scores and the budget for token retention are optimized without the need for additional computational paths.
The Context-Aware Router and Adaptive Ratio Policy further enhance this process by adapting retention strategies based on the specific requirements of each layer and timestep.