NASA’s new dark energy space telescope can also detect killer asteroids
technologyreview.com·3h ago
AI Research
arXiv CS.AI·May 21·6 min read
DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents
TL;DR
Testing LLM-generated software with graphical interfaces (GUIs) is hard because a single failed test run doesn't reveal whether the software itself is broken or the test execution failed. DiagEval is a diagnosis framework that uses trajectory conditioning—replaying execution paths through the UI state-transition graph (the map of possible screen states and transitions)—to distinguish genuine software bugs from evaluator errors.
✦ Why It Matters
Engineers can now confidently evaluate GUI-based LLM applications by pinpointing whether failures stem from the generated code or the test harness itself.
Key Takeaways
- Use trajectory-conditioned diagnosis to disambiguate software bugs from evaluation environment errors in GUI testing.
- Model GUI evaluation as a graph reachability problem across latent UI state-transition graphs, not single paths.
- Apply DiagEval when testing LLM-generated interactive software to improve failure diagnosis accuracy.
- Distinguish evaluator-side execution errors from genuine software defects to reduce false negatives in testing.
Full Summary
Testing LLM-generated interactive software requires running the code against actual GUI environments, not just static code analysis. The core challenge is that correctness depends on reachable properties across an entire state-transition graph (the set of all possible UI states and transitions), but evaluators observe only a single execution trajectory (one path through that graph).
When a test fails, it remains unclear whether the failure stems from a bug in the generated software or from an error in the evaluation setup itself. DiagEval addresses this by using trajectory-conditioned diagnosis—analyzing execution paths within the latent UI state-transition graph to pinpoint failure sources.
This technique enables evaluators to distinguish genuine software defects from evaluation-side execution errors, improving reliability of LLM-generated interactive software assessment.