NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·5 min read
Learning to Reason Efficiently with A* Post-Training
TL;DR
Large language models (LLMs) often struggle with deductive reasoning, leading to incorrect inferences. A* post-training was applied to Llama-3.2 models, utilizing supervised fine-tuning and reinforcement learning to enhance reasoning efficiency.
✦ Why It Matters
Engineers can leverage A* post-training techniques to improve the reasoning capabilities of LLMs in their applications.
Key Takeaways
- Explore A* post-training to enhance LLM reasoning.
- Utilize supervised fine-tuning on execution traces for better accuracy.
- Implement reinforcement learning with A*-informed rewards.
- Consider using imperfect heuristics for larger search spaces.
Full Summary
Deductive reasoning is crucial for many applications of large language models (LLMs), yet these models frequently generate incorrect or redundant inference steps. To address this, a search-based approach was adopted, framing natural language inference as a problem where the goal is to produce valid proofs.
The authors implemented A* post-training on Llama-3.2 models, employing two techniques: supervised fine-tuning on A* execution traces and reinforcement learning with A*-informed reward models. Results showed that models in the 1B–3B parameter range improved from near-zero accuracy to outperforming the larger DeepSeek-V3.2 model.
Notably, while simple correctness rewards maximized accuracy, A*-informed signals provided a balance between accuracy and efficiency. Additionally, models trained with imperfect heuristics performed better in larger search spaces.
These findings suggest a promising direction for enhancing reasoning in LLMs using principles from classical search algorithms.