IronSpec: Increasing the Reliability of Formal Specifications

The guarantees of formally verified systems are only as strong as their trusted specifications (specs). As observed by previous studies [22,52], bugs in formal specs invalidate the assurances that proofs provide. Unfortunately, specs-by their very nature-cannot be proven correct. Currently, the only way to identify spec bugs is by careful, manual inspection.

In this paper we introduce IronSpec, a framework of automatic and manual techniques to increase the reliability of formal specifications. IronSpec draws inspiration from classical software testing practices, which we adapt to the realm of formal specs. IronSpec facilitates spec testing with automated sanity checking, a methodology for writing Spec-Testing Proofs (STPs), and automated spec mutation testing.

We evaluate IronSpec on 14 specs, including six specs of real-world verified codebases. Our results show that IronSpec is effective at flagging discrepancies between the spec and the developer's intent, and has led to the discovery of ten specification bugs across all six real-world verified systems.