BenGQL: An Extensible Benchmarking Framework for Automated GraphQL Testing

GraphQL APIs provide a unified endpoint for retrieving and uploading data in a web application. Due to its efficient data-fetching strategy, which allows for the retrieval of only the required data, GraphQL is gaining popularity. Its software nature necessitates robust testing, both functional and non-functional. As automated testing tools, including load testers and fuzzers, are developed to assess GraphQL APIs, they lack a common set of case studies for rigorous evaluation and comparison.To address this gap, we present BenGQL, a benchmarking framework containing 23 representative open-source GraphQL server applications, spanning different underlying engines and schema complexities. BenGQL provides an extensible infrastructure for running testing tools against these case studies, enabling developers and researchers to: (i) execute testing tools against the same case studies, (ii) analyse and compare results using custom analysis modules, and (iii) extend the benchmark with new case studies, tools, or metrics.The ultimate goal of BenGQL is to foster more rigorous, reproducible research in automated GraphQL testing by providing both the case studies and the infrastructure for running experiments. As a consequence, we release the source code at https://github.com/marcellomaugeri/BenGQL, inviting other researchers to contribute. A video demonstration is also available at https://youtu.be/wZ06Xxa_Koo.