DME-Driver: Integrating Human Decision Logic and 3D Scene Perception in Autonomous Driving

In the field of autonomous driving, two important features of autonomous driving car systems are the explainability of decision logic and the accuracy of environmental perception. This paper introduces DME-Driver, a new autonomous driving system that enhances the performance and reliability of autonomous driving system. DME-Driver utilizes a powerful vision language model as the decisionmaker and a planning-oriented perception model as the control signal generator. To ensure explainable and reliable driving decisions, the logical decision-maker is constructed based on a large vision language model. This model follows the logic employed by experienced human drivers and makes decisions in a similar manner. On the other hand, the generation of accurate control signals relies on precise and detailed environmental perception, which is where 3D scene perception models excel. Therefore, a planning oriented perception model is employed as the signal generator. It translates the logical decisions made by the decision-maker into accurate control signals for the self-driving cars. To effectively train the proposed model, a new dataset for autonomous driving was created. This dataset encompasses a diverse range of human driver behaviors and their underlying motivations. By leveraging this dataset, our model achieves high-precision planning accuracy through a logical thinking process. Recently, deep learning-based methods have achieved remarkable success in the realm of autonomous driving [13, 17, 28, 34, 39] . Some works [11, 24, 20, 26] proposed planning-oriented autonomous driving systems that can be trained end-to-end. As illustrated in Fig. 1 (a), this system [11] encompasses several critical perception modules, including tracking, mapping, motion, and occupancy detection. The outputs from these modules are fed into a planner, which then generates the control signals for the vehicle. This approach leverages the full potential of perception * Corresponding author Preprint. Under review.