Third-party cyber evaluations involving OpenAI models
openai.com·13h ago
AI Research
arXiv CS.AI·Jun 9·5 min read
TempoBench: Evaluating Temporal Causal Reasoning in Large Language Models
TL;DR
Large language models (LLMs) excel at predicting outcomes from inputs but struggle with identifying the minimal causal inputs that lead to those outcomes. TempoBench was developed to evaluate temporal causal reasoning, specifically through tasks like trace simulation (SIM).
✦ Why It Matters
Engineers can improve LLMs by focusing on enhancing their temporal causal reasoning capabilities.
Key Takeaways
- Incorporate TempoBench for evaluating causal reasoning in LLMs.
- Focus on improving minimal causal attribution performance.
- Address overspecification errors in model outputs.
- Utilize fine-tuning on TempoBench to enhance generalization.
How It Works
TempoBench evaluates LLMs through two distinct tasks: SIM, which requires simulating system behavior based on inputs, and MIN, which focuses on identifying the minimal set of inputs that cause a specific outcome. By using Mealy machines, the benchmark ensures that the complexity of the tasks can be controlled and that the causal relationships are accurately labeled.