Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 28·4 min read
One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
TL;DR
Researchers identified a gap in LLM evaluation benchmarks. They built a synthetic dataset with 10k adversarial prompts targeting reasoning failures.
✦ Why It Matters
Use this benchmark to audit LLM robustness before deploying in production reasoning pipelines.
Key Takeaways
- Adopt Layerwise Learning Rates for better model training efficiency.
- Utilize heavy-tailedness to optimize learning rates per layer.
- Expect significant improvements in zero-shot performance metrics.
- Transfer learning rate settings easily from uniform baselines.
How It Works
LLR assigns distinct learning rates to each layer of a Transformer model based on their heavy-tailedness, which reflects the distribution of weight correlations. Layers with less heavy-tailedness are trained faster with higher learning rates, while those with more heavy-tailedness are trained more conservatively with lower rates.
This tailored approach balances the training process across layers, leading to improved convergence and generalization.