NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 20·5 min read
Generative-Evaluative Agreement: A Necessary Validity Criterion for LLM-Enabled Adaptive Assessment
TL;DR
A challenge exists when large language models (LLMs) generate assessment items and score them, leading to a self-referential validation loop. Generative-Evaluative Agreement (GEA) was introduced as a validity criterion to assess whether an LLM's scoring accurately reflects the skill levels it was designed to evaluate.
✦ Why It Matters
Engineers can use GEA to enhance the validity of LLM-generated assessments in educational applications.
Key Takeaways
- Implement GEA to validate LLM scoring in assessments.
- Explore improvements to LLMs for better skill level recovery.
- Consider the implications of self-referential validation in AI systems.
- Use findings to inform the design of adaptive learning tools.
Full Summary
Adaptive assessments are designed to adjust the difficulty of questions based on a student's performance, but when a large language model (LLM) generates both the questions and scores the responses, it creates a self-referential validation issue. Generative-Evaluative Agreement (GEA) was developed to measure the alignment between the LLM's scoring function and the skill levels it was instructed to produce.
In a study, GEA was applied to a two-stage adaptive assessment, where the LLM generated questions and simulated student responses. The results showed that the model was able to recover approximately 50% of the intended skill levels, indicating a significant but incomplete alignment.
This measurement provides a new way to validate LLMs in educational contexts, highlighting the need for further refinement. The findings suggest that while LLMs can be effective in generating and scoring assessments, their accuracy in reflecting true skill levels requires ongoing evaluation and improvement.