Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Aug 6·5 min read
When Absence Is Evidence: Evaluating Completeness-Sensitive Negative Reasoning in Large Language Models
TL;DR
CROWN-QA introduces a framework to evaluate how large language models (LLMs) handle negative reasoning when evidence is incomplete. Results show that LLMs struggle with distinguishing justified negative answers from cases of insufficient evidence, leading to inconsistent reasoning.
✦ Why It Matters
Evaluate your LLM's reasoning capabilities to ensure accurate negative responses in your applications.
Key Takeaways
- Test LLMs with CROWN-QA to assess negative reasoning accuracy.
- Be aware of over-closure issues in LLM responses.
- Use structured prompts to improve evidence evaluation.
- Monitor model performance across different document types.
How It Works
CROWN-QA evaluates LLMs by controlling the coverage of evidence presented to them. CROWN-Synth fixes the question and facts while varying the completeness of evidence, allowing for precise measurement of how well models can discern justified negative answers from insufficient evidence.
CROWN-Real tests these models on actual documents, revealing how they handle real-world complexities.