Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·10d ago·5 min read
What Makes Something Hard(er)? Explaining Question Difficulty in Natural Language
TL;DR
A data-driven approach was developed to explain question difficulty in natural language, revealing actionable insights that improve difficulty predictions. This method outperforms existing models by identifying causal factors behind question difficulty.
✦ Why It Matters
Implement this method to enhance your question design and improve model evaluation today.
Key Takeaways
- Utilize Item Response Theory for estimating question difficulty.
- Generate hypotheses to explain differences in question difficulty.
- Modify questions based on hypotheses to adjust difficulty.
- Combine generated hypotheses with existing models for better predictions.
How It Works
The method begins by estimating question difficulty using responses from a large pool of LLMs through Item Response Theory. It then samples sets of easy and hard questions, prompting an LLM to generate candidate explanations for their difficulty differences.
These explanations are validated against held-out questions to ensure their predictive power.