Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 27·8 min read
Why LLMs Hallucinate on Structured Knowledge: A Mechanistic Analysis of Reasoning over Linearized Representations
TL;DR
LLMs produce hallucinated outputs when reasoning over structured knowledge (graphs, tables) converted into sequential text, even when correct information is available. Researchers conducted a mechanistic analysis revealing hallucinations stem from systematic internal dynamics rather than random errors.
✦ Why It Matters
Understanding hallucination mechanisms in structured reasoning enables targeted fixes to improve LLM reliability on knowledge-intensive tasks.
Key Takeaways
- Focus on improving attention mechanisms in LLMs.
- Enhance grounding techniques for structured knowledge.
- Monitor for semantic grounding failures in outputs.
- Utilize findings to detect hallucinations in various formats.
How It Works
The study reveals that LLMs' attention mechanisms often prioritize shortcut-like structural cues, leading to incomplete context understanding. Additionally, feed-forward layers fail to effectively ground the knowledge, causing the model to rely on its internal memory instead of the provided structured information.