Lune

ASE2025Top-tier venue

Seeing is Fixing: Cross-Modal Reasoning with Multimodal LLMs for Visual Software Issue Repair

Kai Huang, Jian Zhang, Xiaofei Xie, Chunyang Chen

Abstract

Large language model (LLM)-based automated program repair (APR) techniques have shown promising results in resolving real-world github issue tasks. Existing APR systems are primarily evaluated in unimodal settings (e.g., SWE-bench), relying solely on textual issue descriptions and source code. However, these autonomous systems struggle to resolve multimodal problem scenarios (e.g., SWE-bench M) due to limitations in interpreting and leveraging visual information. In multimodal scenarios, LLMs need to rely on visual information in the graphical user interface (GUI) to understand bugs and generate fixes. To bridge this gap, we propose GUIRepair, a cross-modal reasoning approach for resolving multimodal issue scenarios by understanding and capturing visual information. Specifically, GUIRepair integrates two key components including Image2Code and Code2Image to enhance fault comprehension and patch validation. Image2Code extracts relevant project documents based on the issue report, then applies this domain knowledge to generate the reproduced code responsible for the visual symptoms, effectively translating GUI images into executable context for better fault comprehension. Code2Image replays the visual issue scenario using the reproduced code and captures GUI renderings of the patched program to assess whether the fix visually resolves the issue, providing feedback for patch validation. We evaluate GUIRepair on SWE-bench M, and the approach demonstrates significant effectiveness. When utilizing GPT-4o as the base model, GUIRepair solves 157 instances, outperforming the best open-source baseline by 26 instances. Furthermore, when using o4-mini as the base model, GUIRepair can achieve even better results and solve 175 instances, outperforming the top commercial system by 22 instances. This emphasizes the success of our new perspective on incorporating cross-modal reasoning by understanding and capturing visual information to resolve multimodal issues.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution cites SWE-bench: Can Language Models Resolve Real-world Github Issues?
  • SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution cites SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
  • SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution cites Automated Program Repair in the Era of Large Pre-trained Language Models
  • SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering cites SWE-bench: Can Language Models Resolve Real-world Github Issues?
  • Repair Ingredients Are All You Need: Improving Large Language Model-Based Program Repair via Repair Ingredients Search cites CoCoNuT: combining context-aware neural translation models using ensemble for program repair
  • Repair Ingredients Are All You Need: Improving Large Language Model-Based Program Repair via Repair Ingredients Search cites Automated Program Repair in the Era of Large Pre-trained Language Models
  • Automated Program Repair in the Era of Large Pre-trained Language Models cites CoCoNuT: combining context-aware neural translation models using ensemble for program repair
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Repair Ingredients Are All You Need: Improving Large Language Model-Based Program Repair via Repair Ingredients SearchJiayi Zhang, Kai Huang, Jian Zhang, Yang Liu et al.ICSE 2026

Builds on52

  1. SWE-bench: Can Language Models Resolve Real-world Github Issues?Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao et al.ICLR 2024 · 2,082 citations
  2. SWE-agent: Agent-Computer Interfaces Enable Automated Software EngineeringJohn Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret et al.NeurIPS 2024 · 2,059 citations
  3. CoCoNuT: combining context-aware neural translation models using ensemble for program repairThibaud Lutellier, Hung Viet Pham, Lawrence Pang, Yitong Li et al.ISSTA 2020 · 325 citations
  4. Automated Program Repair in the Era of Large Pre-trained Language ModelsChunqiu Steven Xia, Yuxiang Wei, Lingming ZhangICSE 2023 · 321 citations
  5. SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software EvolutionYuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux et al.NeurIPS 2025 · 291 citations

Related papers

  1. Debugging Engine Enhanced by Prior Knowledge: Can We Teach LLM How to Debug?Kunyi Li, Sai Wu, Xiu Tang, Chang Yao et al.FSE 2026 · 1 citation
  2. SWE-Swiss: A Multi-Task Fine-Tuning and RL Recipe for High-Performance Issue ResolutionZhenyu He, Qingping Yang, Wei Shen, Xiaojian Zhong et al.ACL Findings 2026
  3. From Historical Patches to Repair Plans: Outcome-Conditioned Reasoning for Repository-Level Program RepairChenglin Li, Yisen Xu, Zehao Wang, Shin Hwei Tan et al.ICML 2026 · 1 citation
  4. OmniGIRL: A Multilingual and Multimodal Benchmark for GitHub Issue ResolutionLianghong Guo, Wei Tao, Runhan Jiang, Yanlin Wang et al.ISSTA 2025
  5. SWE-Fixer: Training Open-Source LLMs for Effective and Efficient GitHub Issue ResolutionChengxing Xie, Bowen Li, Chang Gao, He Du et al.ACL Findings 2025 · 10 citations
Dusk over the sea between two cliffs drawn in fine vertical lines