NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 26·5 min read
Empirical Analysis and Detection of Hallucinations in LLM-Generated Bug Report Summaries
TL;DR
Large Language Models (LLMs) often generate bug report summaries that include misleading or fabricated information, known as hallucinations. This study developed a section-aware hallucination detection approach that identifies and classifies these inaccuracies in bug report summaries.
✦ Why It Matters
Engineers can leverage section-aware hallucination detection to improve the accuracy of LLM-generated bug reports.
Key Takeaways
- Implement section-aware detection in LLM outputs for better accuracy.
- Train models using synthetic hallucination data for robust evaluation.
- Analyze common hallucination patterns to improve model training.
- Enhance trust in automated tools by reducing hallucination rates.
Full Summary
LLMs are increasingly used to summarize software bug reports, but they frequently produce hallucinations—convincing yet unsupported claims that can mislead developers. An exploratory study revealed that nearly 48% of summaries had missing information, and 12% contained fabricated content.
To address this, a section-aware hallucination detection approach was developed, which predicts the presence of hallucinations, identifies affected sections, and classifies the types of hallucinations. Using the BugsRepo dataset from Mozilla OSS projects, controlled synthetic hallucination injection was employed to create a benchmark for evaluation.
Experimental results demonstrated that the proposed method achieved a report-level Macro-F1 score of 0.89, a section-level Macro-F1 score of 0.83, and a hallucination-type Macro-F1 score of 0.84. The findings emphasize the need for systematic analysis of hallucinations to enhance the reliability of LLM-generated bug report summaries in software maintenance workflows.