DeFT: Maintaining Determinism and Extracting Unit Tests for Autonomous Driving Planning

An Autonomous Driving System (ADS) is a complex software system often composed of multiple modules, each responsible for its own set of tasks. The ADS planning module is responsible for planning the autonomous vehicle’s future driving trajectories and has historically been the most buggy ADS module. In recent years, many approaches have been proposed to test an ADS in complex virtual scenarios through simulation, and these scenarios have been effective in revealing the ADS’s suboptimal decisions. However, due to the randomness of events that occur during the real-time execution of an ADS, test scenarios tend to produce varying outcomes and, in turn, make ADS testing non-deterministic, flaky, and unpredictable. To address this challenge, we propose and evaluate DeFT, an approach that extracts deterministic test cases for the ADS planning module from non-deterministic system-level scenario tests. DeFT monitors the messages exchanged by ADS modules during the execution of system-level scenario tests and reconstructs inputs to reproduce the planning module’s execution. By using DeFT, we find that planning module tests can (1) more accurately reproduce planning module executions that occurred during system-level scenario tests, (2) be used to deterministically detect the same 658 collision failures revealed by system-level scenario tests, and (3) reduce the time needed to reproduce failures by 43.69% to 77.64%.