Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 9·5 min read
Equivariance Breaks the Learning Rate
TL;DR
Equivariant networks trained with Adam can benefit from block normalization, which addresses learning rate mismatches. This adjustment makes Adam competitive with the Muon optimizer across various datasets.
✦ Why It Matters
Implement block normalization in your equivariant network training to enhance Adam's performance today.
Key Takeaways
- Use block normalization to improve Adam's performance on equivariant networks.
- Tune Adam's momentum coefficients independently for better results.
- Evaluate optimizer performance on specific datasets to choose the best tool.
- Consider matrix-structured optimizers like Muon for complex architectures.
How It Works
In equivariant networks, each irrep block learns a shared channel-mixing matrix, leading to a rank limitation in gradient updates. Adam's individual weight rescaling fails to account for this shared structure, causing mismatched learning rates across blocks.
By normalizing updates for each block, the authors ensure consistent scaling, improving the optimizer's effectiveness without introducing new hyperparameters.