NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 25·8 min read
Model Collapse as Cultural Evolution
TL;DR
Large language models degrade when trained on their own outputs (model collapse), but prior work lacked explanation for which linguistic structures fail first. Researchers applied iterated learning theory from cultural evolution to predict degradation patterns, testing LLaMA-2-7B and Mistral-7B across 10 self-training generations in three languages.
✦ Why It Matters
Engineers can now apply task-grounded filtering and understand compositionality dynamics to design self-training pipelines that avoid model degradation.
Key Takeaways
- Apply iterated learning theory to understand model collapse.
- Use task-grounded filtering to sustain compositionality in LLMs.
- Conduct self-training experiments across multiple languages.
- Monitor model behavior to align with human data patterns.
How It Works
The study utilizes iterated learning theory, which posits that knowledge is transmitted and transformed through generations, to explain the non-linear trajectory of compositionality in LLMs. By analyzing the effects of self-training on LLaMA-2-7B and Mistral-7B, the researchers identified that compositionality improves initially but declines later, indicating a complex interaction between model training and data filtering.