PropCov: Effective Coverage Reporting for Property-Based Testing

Property-based testing (PBT), introduced by Haskell’s Quickcheck, is becoming more popular with successful ports for other languages, such as Java’s junit-quickcheck. With PBT, developers write a property test and a data generator. The data generator takes a source of non-determinism and uses it to output well-formed data. The property test exercises the System Under Test (SUT) using the random well-formed data from the generator to ensure a particular property always holds (e.g., data serialized and deserialized should be equal to the original data). The PBT framework then performs many trials, each generating fresh data and executing the property test. A test failure shows a bug to developers, typically in edge-cases. A passing test gives some assurance on the quality of the SUT with regards to the property being tested. Unfortunately, well-known test coverage tools that are instrumental for understanding unit testing work poorly for PBT. In this paper, we present PropCov, a tool for understanding statement coverage in PBT that also provides suggestions for coverage improvement. PropCov employs a novel combination of static analysis with PBT to approximate the maximum possible statement coverage, providing an effective measure of the PBT coverage and making suggestions to developers of where to improve existing tests. PropCov features an easily extensible architecture designed to support new languages, build systems, and PBT frameworks. We evaluated PropCov using 25 Java projects using junit-quickcheck or jqwik, totaling 293 properties, and found that existing tools report missed coverage that is impossible to reach (86% of lines that JaCoCo reports as not covered), which leads developers to consider hundreds of extra lines of code (2897). Unlike existing coverage tools, PropCov results are accurate — only 6.4% of all properties contain unfeasible code, and PropCov only misses 3% of reachable code. Using PropCov’s suggestions, we increased the coverage of 42 tests over 7 projects and found 5 new bugs in 4 projects.