Agentic data science with the Data Agent Kit
cloud.google.com·6h ago
AI Research
arXiv CS.AI·Sep 2·5 min read
Does Fault Localization Beat a Fresh Attempt? A Placebo-Controlled Study of Test-Guided Code Repair
TL;DR
A study comparing fault localization and blind resampling in test-guided code repair found that localization often underperformed. Only 9% of failing candidates provided usable localization, and localized edits were less effective than random attempts.
✦ Why It Matters
Consider using blind resampling for code repair instead of relying on fault localization.
Key Takeaways
- Only 9% of failing candidates provided usable fault localization.
- Localized edits were less effective than blind resampling.
- Random attempts showed better success rates than localized infilling.
- Evaluate the effectiveness of your code repair strategies critically.
Full Summary
Fault localization aims to identify specific code segments responsible for test failures, potentially improving code repair. This study evaluated three approaches: blind whole-solution resampling, spectrum-based localization, and random code span infilling, using three large language models on 488 failing candidates.
Results showed that only 9% of candidates had usable localization data. Among the localizable cases, localized edits performed poorly compared to blind resampling, with a significant loss ratio of 3:40.
Although localized infilling showed some advantage over random attempts, this was not consistently replicated across models. The findings suggest that relying on fault localization may not yield better repair outcomes than random attempts, challenging existing assumptions in automated code repair.