Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 19·5 min read
A decodability criterion predicts when hidden-state selection beats majority voting in large language models
TL;DR
CASE (Correctness-Axis SElection) is a new method for combining answers from large language models that outperforms traditional majority voting by selecting the most accurate candidate based on a decodability measure. This approach shows significant accuracy improvements, particularly on medium and hard questions.
✦ Why It Matters
Evaluate your model's decodability to determine if hidden-state selection is a better choice than majority voting.
Key Takeaways
- Implement CASE to improve answer selection in LLMs.
- Use decodability to assess when to trust hidden-state selection.
- Expect significant accuracy gains on medium and hard questions.
- Evaluate your model's aligned knowledge for better performance.
How It Works
CASE operates by training a linear gate on the hidden states of answer tokens, allowing it to rank candidates based on their likelihood of correctness. The decodability measure quantifies this ranking ability, ensuring that the model can effectively distinguish between correct and incorrect answers without bias from question identity.