Lune

ICSE2026Top-tier venue

LLM-based Agents for Automated Bug Fixing: How Far Are We?

Xiangxin Meng, Zexiong Ma, Pengfei Gao, Chao Peng

Abstract

Large language models (LLMs) and LLM-based Agents have been applied to fix bugs automatically, demonstrating the capability in addressing software defects by engaging in development environment interaction, iterative validation and code modification. However, systematic analysis of these agent systems remain limited, particularly regarding performance variations among top-performing ones. In this paper, we examine six repair systems on the SWEbench Verified benchmark for automated bug fixing. We first assess each system's overall performance, noting the instances solvable by all or none of these systems, and explore the capabilities of different systems. We also compare fault localization accuracy at file and code symbol levels and evaluate bug reproduction capabilities. Through analysis, we concluded that further optimization is needed in both the LLM capability itself and the design of Agentic flow to improve the effectiveness of the Agent in bug fixing.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SWE-bench: Can Language Models Resolve Real-world Github Issues? cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
  • SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering cites SWE-bench: Can Language Models Resolve Real-world Github Issues?
  • SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering cites Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation
  • Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation cites Automated Program Repair in the Era of Large Pre-trained Language Models
  • Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair cites SWE-bench: Can Language Models Resolve Real-world Github Issues?
  • Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair cites SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
  • Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair cites Automated Program Repair in the Era of Large Pre-trained Language Models
  • Automated Program Repair in the Era of Large Pre-trained Language Models cites CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue RepairKai Huang, Jian Zhang, Xiaofei Xie, Chunyang ChenASE 2025 · 5 citations

Builds on27

  1. CodeBERT: A Pre-Trained Model for Programming and Natural LanguagesZhangyin Feng, Daya Guo, Duyu Tang, Nan Duan et al.EMNLP Findings 2020 · 2,828 citations
  2. Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code GenerationJiawei Liu, Chunqiu Steven Xia, Yuyao Wang, Lingming ZhangNeurIPS 2023 · 2,317 citations
  3. SWE-bench: Can Language Models Resolve Real-world Github Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao et al.ICLR 2024 · 2,082 citations
  4. SWE-agent: Agent-Computer Interfaces Enable Automated Software EngineeringJohn Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret et al.NeurIPS 2024 · 2,059 citations
  5. Automated Program Repair in the Era of Large Pre-trained Language ModelsChunqiu Steven Xia, Yuxiang Wei, Lingming ZhangICSE 2023 · 321 citations

Related papers

  1. Understanding Automated Program Repair Agents through the Lens of Traceability: An Empirical StudyIra Ceka, Hailie Mitchell, Saurabh Pujar, Luca Buratti et al.ISSTA 2026
  2. PATCHAGENT: A Practical Program Repair Agent Mimicking Human ExpertiseZheng Yu, Ziyi Guo, Yuhang Wu, Jiahao Yu et al.USENIX Security 2025
  3. SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code AgentsNiels Mündler, Mark Niklas Müller, Jingxuan He, Martin T. VechevNeurIPS 2024 · 172 citations
  4. RepairAgent: An Autonomous, LLM-Based Agent for Program RepairIslem Bouzenia, Premkumar T. Devanbu, Michael PradelICSE 2025 · 54 citations
  5. Taming System Complexity: Demystifying Software Engineering Agents in Diagnosing Linux Kernel FaultsZhenhao Zhou, Zhuochen Huang, Yike He, Chong Wang et al.ACL 2026 · 5 citations
Dusk over the sea between two cliffs drawn in fine vertical lines