Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·24d ago·5 min read
Riemannian--Lorentz Fusion of Vision Transformers and State-Space Models
TL;DR
A novel Riemannian--Lorentz Parameter Fusion (RLPF) method merges Vision Transformers and state-space models without retraining, achieving significant accuracy improvements on benchmark datasets.
✦ Why It Matters
Explore RLPF for merging models in your projects to enhance performance without extensive retraining.
Key Takeaways
- Consider RLPF for merging different model architectures.
- Utilize pre-trained models to save on training costs.
- Experiment with geometry-aware methods for improved accuracy.
- Fine-tune merged models for optimal performance on specific tasks.
Full Summary
Deep learning faces challenges like data exhaustion and high training costs, prompting interest in model merging, which combines pre-trained models without gradient descent. Traditional merging methods struggle with models that have different architectures, such as Vision Transformers (ViTs) and state-space models (SSMs).
The proposed RLPF method aligns parameter groups semantically and projects them into a common coordinate system, utilizing the Lorentz hyperboloid model for hyperbolic space. This approach computes a regularized geodesic barycenter and decodes the results back into the original model branches.
The fine-tuned RLPF system achieved 82.37% accuracy on CIFAR-10, 75.04% on Oxford-IIIT Pet, and 78.58% on ImageNet-1K, outperforming the best parent models. These results highlight the potential of geometry-aware heterogeneous fusion, although RLPF requires training for its gate and final models.