Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·May 25·8 min read
Extracting Search Trees from LLM Reasoning Traces Reveals Myopic Planning
TL;DR
Large language models claim to reason through problems using chain-of-thought (extended text showing their thinking), but it's unclear whether this reasoning constitutes genuine planning. Researchers developed a method to extract and analyze search trees (decision structures showing explored options) from LLM reasoning traces in four-in-a-row, then compared them to human planning patterns.
✦ Why It Matters
Engineers can identify specific limitations in LLM planning and design interventions targeting shallow decision-making rather than assuming extended reasoning improves performance.
Key Takeaways
- LLM planning relies on shallow search breadth, not deep lookahead, despite generating deep reasoning traces.
- Extract and analyze search trees from reasoning outputs to diagnose planning failures in strategic tasks.
- Use causal interventions (selectively removing reasoning steps) to identify which reasoning actually drives decisions.
- Design LLM planning systems to explicitly incorporate deep lookahead mechanisms, not just extended reasoning text.
Full Summary
When large language models generate extended reasoning (chain-of-thought), they appear to deliberate over future outcomes, yet whether this constitutes genuine planning remains unclear. Researchers introduced a method to extract search trees—hierarchical structures representing explored decision branches—from LLM reasoning traces in the four-in-a-row board game.
By fitting computational models to these extracted trees, they characterized how plans are structured and influence move selection. Key findings: LLM search is shallower than human search, performance correlates with breadth (number of options explored) rather than depth (how far ahead they look), and critically, move choices are best explained by myopic models that ignore deep nodes despite LLMs explicitly generating them.
Causal intervention experiments selectively removing reasoning paragraphs confirmed that shallow nodes drive decisions. This contrasts sharply with human expertise, where deeper search predicts performance, revealing a fundamental dissociation between LLM and human planning strategies.