NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
Anytime Training with Schedule-Free Spectral Optimization
TL;DR
Neural network training typically requires predefined learning-rate schedules (rules controlling how fast the model learns over time), forcing costly retuning when data availability changes. SF-NorMuon, a schedule-free spectral optimizer, eliminates this requirement by automatically adapting during training.
✦ Why It Matters
Engineers can train large models without predefined schedules, extracting usable checkpoints anytime while matching hand-tuned optimizer performance.
Key Takeaways
- SF-NorMuon eliminates need to predetermine training duration, reducing retuning overhead.
- Single hyperparameter configuration matches tuned AdamW on 125M–772M parameter models.
- Extract high-quality checkpoints at any point without committing to fixed horizon.
- Weight decay at fast iterate is theoretically essential for long-horizon stability.
Full Summary
Standard neural network training relies on learning-rate schedules—predetermined rules that adjust how quickly the model learns—tied to a fixed training horizon, creating strong path dependence and requiring expensive retuning when conditions change. Schedule-free methods remove explicit schedules, but SF-AdamW, the previous state-of-the-art anytime optimizer (one that works at any stopping point), consistently underperformed well-tuned baselines.
SF-NorMuon, a schedule-free spectral optimizer using spectral methods (techniques analyzing the mathematical properties of weight matrices), closes this performance gap with a single hyperparameter configuration. Tested on language models with 125M and 772M parameters across 1–8× Chinchilla horizons (standard training duration benchmarks), SF-NorMuon matched or exceeded tuned AdamW performance.
Theoretically, the authors proved stationarity guarantees (convergence properties) for schedule-free spectral dynamics and identified weight decay at the fast iterate as essential for long-horizon stability. This enables practitioners to extract high-quality checkpoints at any training point without committing to a horizon in advance, advancing toward truly open-ended continual learning.