Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·25d ago·5 min read
Decodability is Not Causality: Dissociating Probe Readouts from Behavioral Drivers via SAE Decomposition
TL;DR
A new diagnostic method using sparse-autoencoder (SAE) decomposition reveals that linear probes may indicate decodability of features without confirming their causal influence on model behavior. This study shows that combining probe data with feature activation statistics can better identify behaviorally causal features.
✦ Why It Matters
Implement SAE decomposition in your model evaluations to better understand feature causality.
Key Takeaways
- Use SAE decomposition to analyze probe features effectively.
- Recognize that probe accuracy does not imply causal influence.
- Combine probe data with feature activation for better insights.
- Evaluate model behavior changes with shared features for accuracy.
How It Works
The study employs sparse-autoencoder (SAE) decomposition to break down a True/False probe into distinct features, allowing for a detailed analysis of which features align with the probe and which are causally relevant to model behavior. By ranking these features based on their alignment and sensitivity to model outputs, the researchers can discern the true drivers of behavior rather than relying solely on geometric interpretations of probe weights.