Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 12·5 min read
Localizing Anchoring Pathways in Language Models
TL;DR
Language models can be influenced by irrelevant numerical anchors, affecting their reasoning. Researchers developed a logit-difference metric to analyze anchoring effects and used circuit localization techniques on Qwen and Llama models.
✦ Why It Matters
Engineers can leverage these insights to improve model robustness against anchoring biases in numerical tasks.
Key Takeaways
- Explore logit-difference metrics for analyzing model behavior.
- Utilize edge-level methods for better signal recovery in models.
- Investigate the impact of training variations on pathway significance.
- Design prompts to minimize anchoring effects in numerical reasoning.
Full Summary
Language models often exhibit anchoring effects, where irrelevant numbers in prompts can skew their judgments in numerical reasoning tasks. To investigate this phenomenon, researchers introduced a logit-difference metric that compares the likelihood of the correct answer to that of the anchor option.
They employed attribution-based circuit localization techniques on 7B-8B Qwen and Llama models, discovering that edge-level methods more accurately capture anchoring signals than node-level methods. Their findings indicate that low- and high-anchor circuits share structural pathways within a model, although the transfer of these pathways between base and instruction-tuned models is inconsistent.
This suggests that post-training adjustments can alter which pathways are most significant for decision-making. Overall, the study provides insights into the internal mechanisms of language models and how they process anchoring-related information.