Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 24·5 min read
Who Do Language Models Think Is Competent? A Mechanistic Analysis of Occupational Bias
TL;DR
A causal framework reveals that language models (LMs) exhibit internal occupational biases based on demographic attributes, even when behavioral biases are not apparent. This study highlights the importance of examining both internal representations and observable outputs to understand model behavior.
✦ Why It Matters
Evaluate your language models for internal biases beyond observable behaviors to ensure fair outcomes.
Key Takeaways
- Implement a causal framework to analyze model biases.
- Assess both internal representations and outputs for fairness.
- Consider demographic factors in model training and evaluation.
- Use steering vectors to understand user expertise representation.
How It Works
The study introduces a causal framework that decomposes occupational bias into two parts: internal representations of competence and observable outputs. By deriving steering vectors, researchers can manipulate these representations to observe their effects on model behavior in specific tasks.