A Temporal Reasoning Benchmarking Framework for LRMs via Difficulty-controlled and Dynamic Test Generation

TL;DR

TRACE is a new benchmarking framework for Large Reasoning Models (LRMs) that uses dynamic test generation to assess temporal reasoning capabilities. It reveals significant flaws in reasoning processes, including a 28% spurious guessing rate in mid-sized models.

Key Takeaways

• Utilize TRACE for accurate benchmarking of LRMs in temporal reasoning tasks.
• Analyze model performance against difficulty metrics to identify weaknesses.
• Be aware of spurious guessing rates in mid-sized models during evaluations.
• Investigate scale-dependent failure modes to improve model architecture.