NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
Closure-Validated Circuit Discovery in Attention Heads: Co-activation Proposes, Ablation Disposes
TL;DR
Interpretability in AI often focuses on groups of components rather than individual units, raising questions about the effectiveness of co-activation statistics in identifying attention-head circuits. A new method was developed that adapts a sparse-autoencoder clustering technique to attention heads, validated through causal ablation.
✦ Why It Matters
Engineers can use this method to better identify and interpret functional groups within neural networks.
Key Takeaways
- Use co-activation statistics to propose attention-head circuits.
- Validate circuit proposals with causal ablation tests.
- Recognize that not all co-activation signals indicate real circuits.
- Apply closure testing to confirm the functionality of discovered circuits.
How It Works
The method involves clustering attention heads based on their co-activation patterns, which are statistical measures of how often different heads activate together. This clustering is then validated through causal ablation, where the performance impact of removing the clustered heads is measured against a control group.
If the performance degrades significantly, it suggests that the heads form a functional circuit.