Effective REST APIs Testing with Error Message Analysis

REST APIs are essential for building modern enterprise systems, but effectively testing them remains challenging, particularly due to difficulties in inferring constraints from specifications. Current testing approaches typically use feedback from HTTP status codes to guide input generation. However, they overlook valuable information available in the accompanying error messages, reducing their effectiveness in exploring the APIs’ input spaces. In this paper, we propose EmRest, a black-box testing approach that leverages error message analysis to enhance both valid and exceptional test input generation for REST APIs. For each operation under test, EmRest first identifies all possible value assignment strategies for each of its input parameters. It then repeatedly applies combinatorial testing to sample test inputs based on these strategies, and statistically analyzes the error messages (of 400-range status code) received to infer and exclude invalid combinations of value assignment strategies (i.e., constraints of the input space). Additionally, EmRest seeks to mutate valid value assignment strategies that are finally identified to generate test inputs for exceptional testing. The error messages (of 500-range status code) received are categorized to identify bug-prone operations, for which more testing resources are allocated. Our experimental results on 16 real-world REST APIs demonstrates the effectiveness of EmRest. It achieves higher operation coverage than state-of-the-art approaches in 50% of APIs, and detects 226 unique bugs undetected by other approaches.