NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
Polymorphism Is Rotation: Operational Mechanistic Interpretability from a Two-Layer Transformer to Pythia-70m
TL;DR
Transformers trained independently can produce the same outputs despite differing internal representations, a phenomenon termed polymorphism. A method using orthogonal Procrustes fitting allows for the transfer of feature dictionaries between models without retraining.
✦ Why It Matters
Engineers can leverage this rotation technique to enhance model interpretability and transferability without retraining.
Key Takeaways
- Explore orthogonal Procrustes fitting for model alignment.
- Investigate the implications of polymorphism in transformer models.
- Utilize rotation techniques to improve model interpretability.
- Consider the impact of training seed variations on model performance.
Full Summary
In machine learning, transformers are models that process data in layers, but independently trained versions can yield the same outputs while having different internal structures, known as polymorphism. The study introduces a technique called orthogonal Procrustes fitting, which aligns the internal representations of these models by applying a uniform random rotation.
This method was validated on a 104k-parameter Dyck-3 transformer and nine independently trained Pythia-70m models. Results showed that decoder-column cosine similarity was high (98%), but traditional metrics failed to capture the underlying alignment.
By applying a single rotation, reconstruction accuracy improved significantly, achieving within 0.025 explained variance of the best possible outcome. The findings suggest that understanding and manipulating these rotations can enhance model interpretability and transferability.
This research opens avenues for further exploration in larger models, potentially improving their operational efficiency.