Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Sep 10·5 min read
Critical initialization destabilizes higher input derivatives in wide scalar-input networks
TL;DR
Critical initialization in wide scalar-input networks can destabilize higher input derivatives, impacting the performance of physics-informed losses and derivative regularization. This study derives mean-field recursions that reveal how variance behaves at criticality, providing insights into network initialization.
✦ Why It Matters
Reassess your network initialization strategies to ensure stability in higher-order derivatives for better model performance.
Key Takeaways
- Focus on initialization to stabilize higher input derivatives.
- Utilize mean-field recursions for understanding variance behavior.
- Consider the impact of activation functions on derivative growth.
- Implement explicit regularity assumptions in residual networks.
How It Works
The study employs a mathematical framework based on joint Gaussianity of finite derivative jets, allowing for the derivation of mean-field recursions that describe the behavior of input derivatives in wide networks. This approach reveals how the variance of derivatives changes with depth and activation curvature, providing a deeper understanding of network dynamics.