Better Automatic Program Repair by Using Bug Reports and Tests Together
Abstract
Automated program repair is already deployed in industry, but concerns remain about repair quality. Recent research has shown that one of the main reasons repair tools produce incorrect (but seemingly correct) patches is imperfect fault localization (FL). This paper demonstrates that combining information from natural-language bug reports and test executions when localizing faults can have a significant positive impact on repair quality. For example, existing repair tools with such FL are able to correctly repair 7 defects in the Defects4J benchmark that no prior tools have repaired correctly. We develop, Blues, the first information-retrieval-based, statement-level FL technique that requires no training data. We further develop RAFL, the first unsupervised method for combining multiple FL techniques, which outperforms a supervised method. Using RAFL, we create SBIR by combining Blues with a spectrum-based (SBFL) technique. Evaluated on 815 real-world defects, SBIR consistently ranks buggy statements higher than its underlying techniques. We then modify three state-of-the-art repair tools, Arja, SequenceR, and SimFix, to use SBIR, SBFL, and Blues as their internal FL. We evaluate the quality of the produced patches on 689 real-world defects. Arja and SequenceR significantly benefit from SBIR: Arja using SBIR correctly repairs 28 defects, but only 21 using SBFL, and only 15 using Blues; SequenceR using SBIR correctly repairs 12 defects, but only 10 using SBFL, and only 4 using Blues. SimFix, (which has internal mechanisms to overcome poor FL), correctly repairs 30 defects using SBIR and SBFL, but only 13 using Blues. Our work is the first investigation of simultaneously using multiple software artifacts for automated program repair, and our promising findings suggest future research in this directions is likely to be fruitful.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
9 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- AutoCodeRover: Autonomous Program Improvement cites SWE-bench: Can Language Models Resolve Real-world Github Issues?
- AutoCodeRover: Autonomous Program Improvement cites A syntax-guided edit decoder for neural program repair
- Automated Program Repair, What Is It Good For? Not Absolutely Nothing! cites Can automated program repair refine fault localization? a unified debugging approach
- Automated Program Repair, What Is It Good For? Not Absolutely Nothing! cites Baldur: Whole-Proof Generation and Repair with Large Language Models
- Automated Program Repair, What Is It Good For? Not Absolutely Nothing! cites A syntax-guided edit decoder for neural program repair
- Baldur: Whole-Proof Generation and Repair with Large Language Models cites Can automated program repair refine fault localization? a unified debugging approach
- Baldur: Whole-Proof Generation and Repair with Large Language Models cites A syntax-guided edit decoder for neural program repair
- Neural Program Repair with Execution-based Backpropagation cites CURE: Code-Aware Neural Machine Translation for Automatic Program Repair
- Neural Program Repair with Execution-based Backpropagation cites A syntax-guided edit decoder for neural program repair
- Rango: Adaptive Retrieval-Augmented Proving for Automated Software Verification cites Automated Program Repair, What Is It Good For? Not Absolutely Nothing!
- Rango: Adaptive Retrieval-Augmented Proving for Automated Software Verification cites Baldur: Whole-Proof Generation and Repair with Large Language Models
- Rango: Adaptive Retrieval-Augmented Proving for Automated Software Verification cites A syntax-guided edit decoder for neural program repair
Cited by top-tier papers9
- SWE-bench: Can Language Models Resolve Real-world Github Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao et al.ICLR 2024 · 2,082 citations
- AutoCodeRover: Autonomous Program ImprovementYuntong Zhang, Haifeng Ruan, Zhiyu Fan, Abhik RoychoudhuryISSTA 2024 · 96 citations
- Baldur: Whole-Proof Generation and Repair with Large Language ModelsEmily First, Markus N. Rabe, Talia Ringer, Yuriy BrunFSE 2023 · 89 citations
- Automated Program Repair, What Is It Good For? Not Absolutely Nothing!Hadeel Eladawy, Claire Le Goues, Yuriy BrunICSE 2024 · 15 citations
- Rango: Adaptive Retrieval-Augmented Proving for Automated Software VerificationKyle Thompson, Nuno Saavedra, Pedro Carrott, Kevin Fisher et al.ICSE 2025 · 4 citations
Builds on11
- CURE: Code-Aware Neural Machine Translation for Automatic Program RepairNan Jiang, Thibaud Lutellier, Lin TanICSE 2021 · 267 citations
- A syntax-guided edit decoder for neural program repairQihao Zhu, Zeyu Sun, Yuan-an Xiao, Wenjie Zhang et al.FSE 2021 · 214 citations
- Neural Program Repair with Execution-based BackpropagationHe Ye, Matias Martinez, Martin MonperrusICSE 2022 · 146 citations
- Fault Localization with Code Coverage Representation LearningYi Li, Shaohua Wang, Tien N. NguyenICSE 2021 · 120 citations
- Can automated program repair refine fault localization? a unified debugging approachYiling Lou, Ali Ghanbari, Xia Li, Lingming Zhang et al.ISSTA 2020 · 99 citations
Related papers
- On the Effectiveness of Unified Debugging: An Extensive Study on 16 Program Repair SystemsSamuel Benton, Xia Li, Yiling Lou, Lingming ZhangASE 2020 · 35 citations
- DEAR: A Novel Deep Learning-based Approach for Automated Program RepairYi Li, Shaohua Wang, Tien N. NguyenICSE 2022 · 91 citations
- Improving Spectrum-Based Localization of Multiple Faults by Iterative Test Suite ReductionDylan Callaghan, Bernd FischerISSTA 2023 · 16 citations
- Improving Fault Localization and Program Repair with Deep Semantic Features and Transferred KnowledgeXiangxin Meng, Xu Wang, Hongyu Zhang, Hailong Sun et al.ICSE 2022 · 77 citations
- A Deep Dive into Large Language Models for Automated Bug Localization and RepairSoneya Binta Hossain, Nan Jiang, Qiang Zhou, Xiaopeng Li et al.FSE 2024 · 60 citations
