Mean--Fluctuation Dynamics at the Edge of Stability

TL;DR

A new model called mean-fluctuation dynamics reveals how gradient descent can oscillate at high learning rates, potentially enhancing generalization in machine learning. This study rigorously connects these dynamics to the stability of learning trajectories in neural networks.

Key Takeaways

• Experiment with higher learning rates to induce oscillations.
• Utilize the mean-fluctuation dynamics model for better trajectory analysis.
• Apply findings to overparametrized neural networks for enhanced performance.
• Conduct numerical experiments to validate theoretical predictions.