Using Active Learning to Train Predictive Mutation Testing with Minimal Data

Mutation testing is a powerful method of evaluating test suite adequacy. Despite growing industry attention, wide-scale application is frequently limited by the high runtime cost of mutation testing. A set of predictive models have been proposed to mitigate this cost issue, intending to replace the actual execution of a mutated program’s test suite with a predicted result of the tests’ outcome. These predictive models ingest static code features, dynamic execution features, or code and documentation text to produce the predictions. Feature-based models can require a large amount of training data and mutants executed by test cases to become operational. We propose active learning-based predictive mutation testing (AL-PMT) as a way to dramatically reduce the amount of training data needed for a performant model. We conduct experiments to compare AL-PMT’s performance with a non-active learning model and find that AL-PMT quickly converges to improved or on-par performance compared to the baseline of the foundational PMT. AL-PMT achieves 98% of its best possible performance in over 80% of examined projects, while observing only 10% of each project’s mutant set kill status. In addition to training in a fraction of the data required for previous models, AL-PMT is organized in a way that is more amenable to a potential industry application scenario. Besides not requiring the building, running and full mutation testing of several other projects or versions for training data, AL-PMT is able to identify challenging mutants and select them for execution. As such, we expand on the coverage metric provided by basic predictive mutation testing with the ability to guide the targeted execution of important mutants and guiding attention of developers to remaining survived ones. This addresses the rarely mentioned cost of human developer time on fixing the findings of mutation testing, rather than just the computational time spent producing the mutants.