Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
When Do Attention Circuits Form? Developmental Trajectories of Capability and Attention-Sink Emergence Across Three 1B-ClassArchitectures
TL;DR
Attention circuits in language models are not uniformly developed, with distinct emergence patterns observed across different architectures. The study utilized a participation-ratio spectral signal and capability-specific selectivity screens to analyze three 1B-class models.
✦ Why It Matters
Engineers can optimize training strategies by understanding the distinct emergence patterns of attention circuits in language models.
Key Takeaways
- Layers 0 and 1 do not produce BOS-classified heads.
- BOS-attractor emergence varies significantly across models.
- Induction-circuit formation precedes BOS-attractor formation by 10-20x in tokens.
- Circuit identification can occur early in training, not requiring final models.
How It Works
The study employs a participation-ratio (PR) spectral signal to analyze attention-head circuits, focusing on their emergence during training. By applying a capability-specific selectivity screen, the researchers can identify different types of attention heads, such as induction and BOS-attractor heads, and track their development across various training checkpoints.