NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 25·8 min read
LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws
TL;DR
Existing scaling laws for large language models assume monotonic improvement with more compute, but fail to explain performance degradation from overtraining and quantization. Researchers developed the Shannon Scaling Law, which models LLM training as noisy information transmission using Shannon-Hartley theorem principles, revealing a fundamental capacity limit.
✦ Why It Matters
Engineers can now predict when scaling becomes counterproductive and optimize model-data tradeoffs to maintain signal quality.
Key Takeaways
- Implement the Shannon Scaling Law for better model predictions.
- Monitor signal-to-noise ratios during LLM training.
- Use perturbation-aware techniques to enhance model robustness.
- Explore the implications of non-monotonic performance in LLMs.
How It Works
The Shannon Scaling Law models LLM training as a communication process, where model parameters represent channel bandwidth and training tokens signify signal power. This analogy allows researchers to analyze how noise impacts learning, emphasizing the importance of maintaining a high signal-to-noise ratio to prevent performance drops.