Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
Detecting Knowledge Gaps from Conversational AI Interactions Using Curriculum Prerequisite Graphs
TL;DR
Students often ask numerous questions to conversational AI teaching assistants, revealing potential knowledge gaps. A few-shot text classifier was developed to map these questions to curriculum topics using a prerequisite knowledge graph.
✦ Why It Matters
Engineers can leverage this approach to identify and address knowledge gaps in educational settings effectively.
Key Takeaways
- Utilize conversational AI logs to identify knowledge gaps.
- Implement few-shot learning techniques for question classification.
- Analyze correlations between question volume and student difficulties.
- Enhance curriculum design based on actionable insights from AI interactions.
Full Summary
In large online courses, students frequently interact with conversational AI teaching assistants, generating valuable data that can highlight knowledge gaps. Researchers created a few-shot text classifier to categorize student questions based on a prerequisite knowledge graph derived from course concepts.
Evaluated on 1,340 questions from 164 graduate students in an AI course, the classifier achieved an accuracy of 80% across 43 labels, including 42 curriculum topics and an 'unknown' category. A significant correlation (rho = 0.491, p = 0.008) was found between the volume of questions on specific topics and students' self-reported difficulties.
These findings suggest that analyzing conversational AI interactions can provide instructors with a structured view of which topics need more attention, ultimately enhancing the learning experience.