Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
When Attribution Patching Lies: Diagnosis and a Second-Order Correction
TL;DR
Attribution patching, a method for understanding language model behavior, suffers from reliability issues due to non-linearities in the model. This work introduces a Hessian-vector-product (HVP) correction that significantly reduces errors with minimal computational overhead.
✦ Why It Matters
Engineers can leverage HVP for more reliable model interpretability without incurring high computational costs.
Key Takeaways
- Use HVP for efficient error correction in model interpretability.
- Implement reliability scores to assess attribution trustworthiness.
- Adopt the Screen-Flag-Fix workflow for targeted computational efforts.
- Evaluate model performance across various parameter sizes for best practices.
Full Summary
Understanding how language models operate requires identifying which internal components influence their behavior, a process known as mechanistic interpretability. Attribution patching is a common technique used for this purpose, but its reliability is questionable due to errors arising from non-linear interactions within the model.
This research introduces a Hessian-vector-product (HVP) correction, which addresses these errors effectively with just one additional backward pass during computation. Evaluations across five model families, ranging from 124 million to 9 billion parameters, demonstrate that HVP not only matches but often exceeds the accuracy of traditional methods like Integrated Gradients while being computationally efficient.
The findings suggest that HVP can enhance circuit recovery in language models, allowing researchers to focus computational resources on unreliable components. This work supports a new workflow called Screen-Flag-Fix, optimizing the interpretability process in machine learning.