Cross-Refactoring-Type Test Program Migration for Refactoring Engines

Refactoring engines are an essential part of modern IDEs, supporting automated or semi-automated software refactorings. However, these refactoring engines suffer from software bugs, just like other complex software systems, and buggy refactoring engines may silently inject fatal errors into software projects. Consequently, thorough testing of refactoring engines is highly desirable. To this end, this paper proposes an automated approach called EngineTest to test refactoring engines. Unlike existing approaches, it is the first in this line to leverage both LLMs and bug reports of refactoring types other than the targeted refactoring type. The key rationale is that the same mistake, e.g., ignoring the potential name shadowing, may result in similar errors in multiple refactoring types.Consequently, we may retrieve a bug report for a refactoring type and carefully construct a few-shot example as input, guiding LLMs to migrate it into test programs for exercising other refactoring types.Finally, we run the test cases and validate them with differential testing and LLM-based inconsistent checking. We evaluate the proposed approach with widely used state-of-the-practice refactoring engines in Eclipse, NetBeans, and IntelliJ IDEA. Our evaluation results demonstrate that a total of 85 previously unknown bugs (including Java, C/C++, and Python) have been identified, and 30 of them have been manually confirmed by the tool vendors.