RBCTest: Leveraging LLMs to Mine and Verify Oracles of API Response Bodies for RESTful API Testing

In API testing, deriving logical constraints on API response bodies to be used as oracles is crucial in generating test cases and performing automated testing of RESTful APIs. However, existing approaches are restricted to dynamic analysis in which oracles are extracted via the execution of APIs as part of the system under test. In this paper, we propose a complementary LLM-based, static approach in which the constraints for API response bodies are mined from API specifications. We leverage large language models (LLMs) to comprehend the API specifications, mine constraints for response bodies, and generate test cases. To reduce LLMs' hallucination, we apply an Observation-Confirmation (OC) scheme which uses initial prompts to contextualize constraints, allowing subsequent prompts to more accurately confirm their presence. Our empirical results show that RBCTest with OC prompting achieves high precision in constraint mining with the average from 85.1%-93.6%. It also performs well in generating test cases from mined constraints, with a precision from 86.4%-91.7%. We also use the test cases generated by RBCTest to detect 46 mismatches between the API specification and actual response data for 19 real-world APIs. Four of the mismatches were, in fact, reported in developers' forums.