Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Jul 27·5 min read
Math Education Digital Shadows for Investigating Learning with GenAI: Mathematics Performance, Anxiety, and Confidence in LLMs
TL;DR
Math Education Digital Shadows (MEDS) is a dataset that evaluates how large language models (LLMs) perform in mathematics and their associated biases. Findings reveal that LLMs display human-like negative attitudes and overconfidence in math tasks, highlighting the need for safer AI tutors.
✦ Why It Matters
Consider using the MEDS dataset to improve AI tutor designs and address biases in math education.
Key Takeaways
- Explore the MEDS dataset for insights on LLM performance in math.
- Address biases in AI tutors by analyzing LLM attitudes towards math.
- Utilize psychometric tools to assess student interactions with AI.
- Develop strategies to mitigate overconfidence in AI-assisted learning.
Full Summary
Math Education Digital Shadows (MEDS) is a comprehensive dataset designed to analyze the mathematical reasoning of large language models (LLMs) like Mistral and Microsoft Phi. It includes 28,000 runs from 14 LLMs, generated under two conditions: human-shadow and AI-assistant.
Each entry, or digital shadow, contains prompts, psychological metadata, and four mathematics tasks, including interviews on math relationships and psychometric questionnaires assessing self-efficacy and anxiety. The analysis indicates that LLMs exhibit significant differences in performance and attitudes, such as negative feelings towards math and logical fallacies, particularly in human-shadow mode.
These insights can inform the development of safer AI tutors and enhance understanding of LLM biases in educational contexts.