Evolutionary improvement of assertion oracles

Assertion oracles are executable boolean expression placed inside the program. A perfect assertion oracle should pass (return true) for all correct executions and fail (return false) for all incorrect executions. The difficulty of manually design or automatically generate perfect oracles often leads to assertions that fail to distinguish between correct and incorrect executions. In other words, they have false positives and false negatives that we call oracle deficiencies. In this paper, we propose GASSERT (Genetic ASSERTion improvement), the first technique to automatically improve assertions oracles. Given an assertion oracle and its oracle deficiencies, GASSERT uses a novel evolutionary algorithm that explores the space of possible assertions to identify a new assertion with fewer oracle deficiencies than the original assertion. Our empirical evaluation on 34 JAVA methods from 7 JAVA code bases shows that GASSERT effectively improves assertion oracles. Moreover, GASSERT outperforms two baselines (unguided-random and invariantbased oracle improvement) and was competitive with and in some cases even outperformed human improved assertions.