RCTrans: Radar-Camera Transformer via Radar Densifier and Sequential Decoder for 3D Object Detection

In radar-camera 3D object detection, the radar point clouds are sparse and noisy, which causes difficulties in fusing camera and radar modalities. To solve this, we introduce a novel query-based detection method named Radar-Camera Transformer (RCTrans). Specifically, we first design a Radar Dense Encoder to enrich the sparse valid radar tokens, and then concatenate them with the image tokens. By doing this, we can fully explore the 3D information of each interest region and reduce the interference of empty tokens during the fusing stage. We then design a Pruning Sequential Decoder to predict 3D boxes based on the obtained tokens and random initialized queries. To alleviate the effect of elevation ambiguity in radar point clouds, we gradually locate the position of the object via a sequential fusion structure. It helps to get more precise and flexible correspondences between tokens and queries. A pruning training strategy is adopted in the decoder, which can save much time during inference and inhibit queries from losing their distinctiveness. Extensive experiments on the large-scale nuScenes dataset prove the superiority of our method, and we also achieve new state-of-theart radar-camera 3D detection results. Our implementation is available at https://github.com/liyih/RCTrans . Introduction 3D object detection is an important perceptual task that can be widely applied in many fields. Existing high-precision detection algorithms often rely on the input of LiDAR(Yan, Mao, and Li 2018; Yin, Zhou, and Krahenbuhl 2021). However, LiDAR is expensive and easily damaged (Lin et al. 2024), which is not conducive to the commercialization of the algorithm. To reduce the practical cost, recent works begin to focus on using low-cost radars and multi-view cameras to improve detection performance. Camera sensors can provide rich texture information, while radar sensors can provide three-dimensional information that is not affected by different weather conditions (Kim et al. 2023a). However, the radar sensor has two inherent drawbacks, i.e., sparsity and noise. Firstly, a radar typically releases