Revisiting Test-Case Prioritization on Long-Running Test Suites

The prolonged continuous integration (CI) runs are a ecting timely feedback to software developers. Test-case prioritization (TCP) aims to expose faults sooner by reordering tests such that the ones more likely to fail are run earlier. TCP is thus especially important for long-running test suites. While many studies have explored TCP, they are based on outdated CI builds from over 10 years ago with test suites that last several minutes, or builds from inaccessible, proprietary projects. In this paper, we present LRTS, the rst dataset of long-running test suites, with 21,255 CI builds and 57,437 testsuite runs from 10 large-scale, open-source projects that use Jenkins CI. LRTS spans from 2020 to 2023, with an average test-suite run duration of 6.5 hours. On LRTS, we study the e ectiveness of 59 leading TCP techniques, the impact of confounding test failures on TCP, and TCP for failing tests with no prior failures. We revisit prior key ndings (9 con rmed, 2 refuted) and establish 3 new ndings. Our results show that prioritizing faster tests that recently failed performs the best, outperforming the sophisticated techniques. CCS Concepts • Software and its engineering → Software testing and debugging; Software reliability.