Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 13·5 min read
Accuracy and Order Sensitivity Diverge Under Label-Free Strategies
TL;DR
Label-free strategies for evaluating large language models were tested to reduce bias from option order in multiple-choice questions. However, these strategies did not consistently improve accuracy, revealing that simply withholding option labels is not sufficient.
✦ Why It Matters
Consider revising evaluation methods for language models to account for order sensitivity in multiple-choice tasks.
Key Takeaways
- Evaluate language models with awareness of order sensitivity in MCQs.
- Test cyclic permutation of options to improve accuracy.
- Consider using generation-then-matching approaches cautiously.
- Reassess the impact of withholding option labels on model performance.
How It Works
The study investigates two strategies to reduce bias in MCQ evaluations. The generation-then-matching approach generates answers before matching them to options, while the isolated scoring method evaluates options independently, avoiding positional influence.
Both methods aim to provide a fairer assessment of model knowledge.