Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
LoRA-Muon: Spectral Steepest Descent on the Low-Rank Manifold
TL;DR
Finetuning deep learning models using Low-Rank Adaptation (LoRA) can be challenging due to sensitivity in optimization settings. LoRA-Muon, a new optimizer, applies spectral steepest descent to improve low-rank training efficiency and learning rate transferability.
✦ Why It Matters
Engineers can leverage LoRA-Muon for more efficient and effective model finetuning in resource-constrained environments.
Key Takeaways
- Use LoRA-Muon for efficient model finetuning.
- Optimize learning rates across different model configurations.
- Avoid QR-decomposition for better memory efficiency.
- Achieve lower validation loss compared to dense training.
Full Summary
Low-Rank Adaptation (LoRA) is a technique that reduces the computational and memory demands of finetuning deep learning models, but it often struggles with optimization sensitivity. LoRA-Muon was developed by applying the Muon optimizer's spectral steepest-descent rule to the low-rank context, enhancing its performance.
This new method includes a split weight-decay rule and allows optimal learning rates to transfer effectively across various model configurations. In a study using the TinyShakespeare dataset, a rank-2 LoRA-Muon configuration matched the best learning rate of dense training, while a rank-32 configuration achieved a lower mean validation loss than the dense baseline.
Additionally, LoRA-Muon avoids the need for QR-decomposition and second moment storage, making it more efficient for hardware accelerators. These findings suggest that LoRA-Muon can be a robust alternative for model finetuning, particularly in scenarios with limited resources.