Research on Test Flakiness: from Unit to System Testing

Test flakiness has been a common problem in the software automation testing, affecting the effectiveness and productivity of test automation. Many studies have been published to tackle test flakiness in the software research community, and existing test automation tools provide capabilities to address issues related to test flakiness. This paper describes our review of recent approaches in the research community and popular industrial tools with capabilities to tackle test flakiness. Our review indicates that while many studies focus on test flakiness in unit testing, few address the problem in the end-to-end system testing. On the contrary, industrial test automation tools tend to be more concerned with test flakiness at the system level.