Agentic data science with the Data Agent Kit
cloud.google.com·7h ago
AI Research
arXiv CS.AI·Aug 17·5 min read
Rethinking Automated Program Repair: The Impact of Bug Complexity, Fault Localization, and LLM Cost-efficiency
TL;DR
This study evaluates the effectiveness of LLM-based Automated Program Repair (APR) techniques, revealing that while bug complexity and fault localization impact performance, over 50% of moderately complex bugs can be repaired using low-cost models.
✦ Why It Matters
Consider using low-cost LLMs for repairing moderately complex bugs in your projects today.
Key Takeaways
- Evaluate bug complexity before selecting an APR technique.
- Use low-cost LLMs for effective bug repairs.
- Consider fault localization precision to improve repair outcomes.
- Analyze cost-efficiency when choosing between different LLMs.
Full Summary
Software bugs pose significant challenges in development, prompting the need for effective Automated Program Repair (APR) techniques. This research investigates how factors like bug complexity, fault localization, and reasoning settings influence the performance of LLM-based APR systems, specifically ChatRepair and CodeCorrector, using models such as DeepSeek, GPT, and Llama.
Through a comprehensive empirical analysis, it was found that while structurally complex bugs and imprecise fault localization hinder repair efforts, LLM-based techniques still demonstrate competitive effectiveness. Notably, the study highlights that over 50% of moderately complex bugs can be repaired by low-cost LLMs.
Additionally, the performance gap between APR techniques widens with less precise fault localization, and higher-cost models do not always guarantee better cost-efficiency. For instance, GPT-5 outperformed DeepSeek models in repairing complex bugs, yet DeepSeek-V3.2 exhibited the best overall cost-efficiency.