Let the Code Speak: Incorporating Program Dynamic State for Better Method-Level Fault Localization

Fault localization (FL) is a critical but time-consuming part of software debugging. With the improvement of the Large Language Models (LLMs) in their code capabilities, the increasing demand for automated software development has encouraged more research on building LLM-based Fault Localization (LLMFL) systems. However, existing LLMFL techniques are typically restricted to predicting bug locations by analyzing static code, while overlooking crucial dynamic program state of the software. This lack of context makes LLMs prone to generating "hallucinations", incorrectly identifying bug-free code as suspicious. To address this, this paper introduces PingFL, the LLMFL system that incorporates program dynamic information for more accurate automatic fault localization. PingFL comprises a Fault Localization (FL) agent and a Print Debugging (PD) agent. The FL agent is tasked with understanding the root cause through a set of callable tools. When the FL agent nominates a location as suspicious, it would entrust the PD agent to verify the suspected issue through multiple rounds of print debugging. In particular, these two agents communicate efficiently by conveying the textual thought generated by the LLM. The evaluation on 812 real-world bugs from the Defects4J benchmark shows that PingFL can localize 450 bugs within Top-1, which significantly outperforms other LLM-based approaches by 41% to 122%. A deeper dive into PingFL’s performance reveals that it exhibits specific FL strategies and tool usage patterns even without explicit instructions. Finally, PingFL proves to be cost-effective, spending an average of $0.23 and 104.62 seconds per bug, with the print debugging mechanism accounting for only$0.07 and 48.14 seconds.