NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·Jun 15·5 min read
The Weight Norm Sets the Grokking Timescale: A Causal Delay Law
TL;DR
Grokking is the phenomenon where neural networks take time to generalize beyond training data. Researchers intervened on the weight norm during training to determine its effect on grokking timescales.
✦ Why It Matters
Engineers can optimize neural network training by adjusting weight norms to control grokking timescales effectively.
Key Takeaways
- Control weight norms to influence grokking timescales.
- Use a fixed multiple of Wc to manage training delays.
- Monitor weight decay effects on generalization.
- Implement LayerNorm to decouple weight scale from function.
How It Works
The study demonstrates that by intervening on the weight norm during training, researchers can directly influence when a neural network begins to generalize. They found that grokking occurs when the weight norm reaches a critical threshold, Wc, which is consistent across different training setups.
By clamping the weight norm to a fixed multiple of Wc, they established a clear exponential relationship between the norm and the grokking delay, allowing for precise control over the training process.