NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·2h ago
AI Research
arXiv CS.AI·May 26·10 min read
LGMT: Logic-Grounded Metamorphic Testing for Evaluating the Reasoning Reliability of LLMs
TL;DR
Large Language Models (LLMs) often show strong performance in logical reasoning, but their reliability is questionable due to reliance on static benchmarks. LGMT (Logic-Grounded Metamorphic Testing) was developed to evaluate LLM reasoning by using first-order logic to create semantically invariant test cases.
✦ Why It Matters
Engineers can use LGMT to improve the reliability of LLMs by identifying and addressing reasoning defects.
Key Takeaways
- Implement LGMT to evaluate LLM reasoning reliability.
- Focus on robustness under logical invariance in assessments.
- Explore symbol-level and conclusion-level variations in LLMs.
- Consider advanced prompting techniques to mitigate reasoning issues.
Full Summary
Large Language Models (LLMs) have demonstrated impressive capabilities in logical reasoning tasks, yet their reliability remains uncertain, primarily because existing evaluations depend on static benchmarks that do not account for variations in logical expressions. To address this gap, LGMT (Logic-Grounded Metamorphic Testing) was introduced as an oracle-free framework that utilizes first-order logic (FOL) to derive metamorphic relations, allowing for the construction of test cases that maintain semantic consistency despite logical transformations.
By checking for consistency across these cases, LGMT effectively identifies reasoning defects in LLMs. Experiments conducted on six state-of-the-art LLMs revealed that LGMT exposed significant hidden flaws that traditional reference-based evaluations overlooked.
Notably, models were found to be particularly sensitive to changes at the symbol and conclusion levels, and advanced prompting techniques like Few-shot Chain of Thought (CoT) only partially alleviated these issues. These findings suggest a shift in LLM evaluation towards assessing robustness under logical invariance rather than isolated correctness, providing a scalable method for diagnosing reasoning failures.