Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·May 27·6 min read
Reasoning, Code, or Both? How Large Language Models Handle Variations in Math Questions
TL;DR
LLMs perform well on math benchmarks but fail when problems change slightly (different numbers or names), revealing poor robustness. Researchers systematically compared natural language reasoning versus code execution methods to determine which approach maintains accuracy across problem variations.
✦ Why It Matters
Engineers can choose code execution over reasoning for math tasks requiring robustness to problem variations.
Key Takeaways
- Use chain-of-thought prompting for better reasoning in math tasks.
- Be cautious with code execution methods for math problem variations.
- Test LLMs on diverse problem sets to evaluate robustness.
- Monitor accuracy drops when modifying problem parameters.
How It Works
Chain-of-thought (CoT) prompting encourages LLMs to articulate their reasoning step-by-step, which helps maintain accuracy even when problems are altered. In contrast, code execution methods like PAL and SBSC generate and run Python code, but this approach did not significantly improve the models' ability to handle variations.