Third-party cyber evaluations involving OpenAI models
openai.com·14h ago
AI Research
arXiv CS.AI·Jun 10·5 min read
SAFE: An LLM-as-Verifier Framework for Evidence-Grounded Multi-Hop Reasoning
TL;DR
Multi-hop question answering (QA) often suffers from models providing correct answers through flawed reasoning. SAFE is a framework that verifies reasoning steps during answer generation by using evidence-grounded Knowledge Graph (KG) triples.
✦ Why It Matters
Engineers can implement SAFE to enhance the accuracy and reliability of multi-hop QA systems.
Key Takeaways
- Implement SAFE to verify reasoning steps in QA systems.
- Use Knowledge Graph triples for evidence-grounded reasoning.
- Focus on stepwise verification to improve model accuracy.
- Train verifiers under KG-grounded constraints for reliability.
Full Summary
Multi-hop question answering (QA) tasks require models to connect multiple pieces of information to arrive at a correct answer, but many large language models (LLMs) achieve this through incorrect reasoning paths. SAFE, or the LLM-as-Verifier framework, addresses this issue by verifying each reasoning step against provided evidence and previous steps, ensuring that the reasoning is valid.
It breaks down reasoning into atomic units represented as Knowledge Graph (KG) triples, which allows for systematic verification. During training, SAFE constructs reliable verifier data under KG-grounded constraints, while at inference, an external verifier checks each generated step for validity.
The implementation of SAFE led to an average accuracy improvement of 8.8 percentage points across three different multi-hop QA benchmarks. These findings suggest that a focus on stepwise reasoning verification can significantly enhance the reliability of LLMs in complex QA tasks.