Instrumentation-Driven Evolution-Aware Runtime Verification

Runtime verification (RV) found hundreds of bugs by monitoring passing tests against formal specifications (specs). RV first instruments a program to obtain relevant events, e.g., method calls, to monitor. A hindrance to RV adoption, especially in continuous integration, is its high overhead. So, prior work proposed spec-driven evolution-aware techniques to speed up RV. They use complex analysis to re-monitor a subset of specs related to code impacted by changes. But, these techniques assume that RV overhead is dominated by monitoring time, and their designs often sacrifice safety (ability to find all new violations) for speed. We present IMOP, the first instrumentation-driven evolution-aware RV framework. IMOP leverages a recent observation that RV overhead during testing is often dominated by instrumentation, not monitoring. IMOP embodies a family of 14 techniques that aim to safely speed up RV by simply re-instrumenting only changed code. Instrumentation from the old revision is re-used for unchanged code, and all specs are re-monitored in the new revision. We implement IMOP as a Maven plugin and evaluate it on 2,028 revisions of 66 projects, using 160 specs of correct JDK API usage. IMOP is safe by design. It is up to 40.2x faster than re-running RV from scratch after each change, and 17.8x and 6.7x faster than safe and unsafe spec-driven techniques, respectively. IMOP is faster than just applying regression test selection to RV.