Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·11d ago·5 min read
Structure vs. Chain-of-Thought: Evaluating LLM Criteria Extraction for Depression Severity
TL;DR
A comparison of two methods for assessing depression severity using large language models (LLMs) reveals that criteria extraction is not significantly better than chain-of-thought prompting. The findings suggest that while criteria extraction can be more auditable, it often misses severe cases.
✦ Why It Matters
Consider using direct rating methods for more accurate detection of severe depression cases in social media analysis.
Key Takeaways
- Evaluate the effectiveness of LLMs for mental health assessments.
- Use direct rating methods to improve detection of severe cases.
- Be cautious of relying solely on criteria extraction for accuracy.
- Consider the context of the data source when applying models.
Full Summary
This study evaluates two approaches for assessing depression severity from social media posts using large language models (LLMs): direct rating and clinical criteria extraction. The researchers tested three LLMs on two Reddit datasets and compared their performance against established questionnaires (PHQ-9 and BDI-II) using quadratic weighted kappa for agreement measurement.
Results showed that criteria extraction only outperformed chain-of-thought prompting when decision thresholds were fitted on labeled data, but overall gains were not significant. Notably, the 9B model frequently labeled posts as severe, regardless of the prompting method, while the a priori rule based on PHQ-9 criteria performed better without labels.
Additionally, the study found that moving from direct prompting to chain-of-thought and then to extraction led to increased misses of severe cases, indicating that higher ordinal agreement does not equate to better detection of severe depression.