NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 21·6 min read
AR1-ZO: Topology-Aware Rank-1 Zeroth-Order Queries for High-Rank LoRA Fine-Tuning
TL;DR
Fine-tuning large language models without storing intermediate activations (zeroth-order optimization) combined with LoRA (Low-Rank Adaptation—compact trainable parameter modules) creates a rank paradox: higher-rank adapters improve capacity but break gradient-free optimization's signal detection. AR1-ZO, a topology-aware rank-1 zeroth-order method, solves this by adapting perturbation strategies to the adapter's structure, enabling effective high-rank LoRA fine-tuning without backpropagation memory overhead.
✦ Why It Matters
Engineers can now fine-tune large models on memory-constrained hardware while maintaining adapter expressiveness without storing activation data.
Key Takeaways
- Use AR1-ZO for memory-efficient fine-tuning when backpropagation storage is infeasible.
- High-rank LoRA adapters now work with zeroth-order methods via topology-aware queries.
- Gradient observability is preserved by respecting adapter structure in perturbation design.
- Achieves comparable performance to standard backprop with significantly lower memory requirements.
Full Summary
Large language models are expensive to fine-tune because backpropagation requires storing intermediate activations in memory. Zeroth-order optimization avoids this by estimating gradients through function evaluations alone, while LoRA reduces trainable parameters by learning low-rank weight updates.
However, combining them creates a fundamental problem: standard two-point zeroth-order methods either perturb a number of coordinates proportional to LoRA rank (expensive) or use coordinate-wise updates that make gradient signals disappear at high ranks. AR1-ZO addresses this by introducing topology-aware rank-1 query strategies that preserve gradient observability across the adapter's structure.
The method maintains computational efficiency while enabling high-rank LoRA fine-tuning without activation storage, making it practical for memory-constrained settings. Results demonstrate that AR1-ZO achieves comparable or better performance than standard backpropagation-based LoRA while reducing memory overhead.