Practical Non-Intrusive GUI Exploration Testing with Visual-based Robotic Arms

Graphical User Interface (GUI) testing has been a significant topic in the software engineering community. Most existing GUI testing frameworks are intrusive and can only support some specific platforms, which are quite limited. With the development of distinct scenarios, diverse embedded systems or customized operating systems on different devices do not support existing intrusive GUI testing frameworks. Some approaches adopt robotic arms to replace the interface invoking of mobile apps under test and use computer vision technologies to identify GUI elements. However, Some challenges remain unsolved with such approaches. First, existing approaches assume that GUI screens are fixed so that they cannot be adapted to diverse systems with different screen conditions. Second, existing approaches use XY-plane robotic arm system, which cannot flexibly simulate human testing operations. Third, existing approaches ignore the compatibility bugs of apps and only focus on the crash bugs. To sum up, a more practical approach is required for the non-intrusive scenario. In order to solve the remaining challenges, we propose a practical non-intrusive GUI testing framework with visual-based robotic arms, namely RoboTest. RoboTest integrates a set of novel GUI screen and widget detection algorithm that is adaptive to detecting screens of different sizes and then to extracting GUI widgets from the detected screens. Then, a complete set of widely-used testing operations are applied with a 4-DOF robotic arm, which can more effectively and flexibly simulate human testing operations. During the app exploration, RoboTest integrates the specially designed Principle of Proximity-guided (PoP-guided) exploration strategy, which chooses close widgets of the previous operation targets to reduce the robotic arm movement overhead and improve exploration * Chunrong Fang is the corresponding author.