Automata-Based Automated Detection of State Machine Bugs in Protocol Implementations

—Implementations of stateful security protocols must carefully manage the type and order of exchanged messages and cryptographic material, by maintaining a state machine which keeps track of protocol progress. Corresponding implementation flaws, called state machine bugs , can constitute serious security vulnerabilities. We present an automated black-box technique for detecting state machine bugs in implementations of stateful network protocols. It takes as input a catalogue of state machine bugs for the protocol, each specified as a finite automaton which accepts sequences of messages that exhibit the bug, and a (possibly inaccurate) model of the implementation under test, typically obtained by model learning. Our technique constructs the set of sequences that (according to the model) can be performed by the implementation and that (according to the automaton) expose the bug. These sequences are then transformed to test cases on the actual implementation to find a witness for the bug or filter out false alarms. We have applied our technique on three widely-used implementations of SSH servers and nine different DTLS server and client implementations, including their most recent versions. Our technique easily reproduced all bugs identified by security researchers before, and produced witnesses for them. More importantly, it revealed several previously unknown bugs in the same implementations, two new vulnerabilities, and a variety of new bugs and non-conformance issues in newer versions of the same SSH and DTLS implementations.