PvNeXt: Rethinking Network Design and Temporal Motion for Point Cloud Video Recognition

Point cloud video perception has become an essential task for the realm of 3D vision. Current 4D representation learning techniques typically engage in iterative processing coupled with dense query operations. Although effective in capturing temporal features, this approach leads to substantial computational redundancy. In this work, we propose a framework, named as PvNeXt, for effective yet efficient point cloud video recognition, via personalized one-shot query operation. Specially, PvNeXt consists of two key modules, the Motion Imitator and the Single-Step Motion Encoder. The former module, the Motion Imitator, is designed to capture the temporal dynamics inherent in sequences of point clouds, thus generating the virtual motion corresponding to each frame. The Single-Step Motion Encoder performs a one-step query operation, associating point cloud of each frame with its corresponding virtual motion frame, thereby extracting motion cues from point cloud sequences and capturing temporal dynamics across the entire sequence. Through the integration of these two modules, PvNeXt enables personalized one-shot queries for each frame, effectively eliminating the need for frame-specific looping and intensive query processes. Extensive experiments on multiple benchmarks demonstrate the effectiveness of our method. INTRODUCTION Point cloud videos serve as a pivotal character, offering a dynamic perspective into our environment, which is fundamental in the realms of robotics and AR systems. These sequences, which present movements within the physical domain, are crucial in delineating environmental transformations and facilitating interactions within said environments. This contrasts starkly with the limited descriptive capabilities of 2D images or static 3D point clouds. Therefore, enhancing the ability of point cloud video perception becomes a significant yet challenging task. However, 4D data representation learning presents vital challenges and remains a nascent field of inquiry. The amalgamation of 3D geometry and dynamic motion often leads to data redundancy within an exceedingly high-dimensional space, which heavily hinders the development of efficient spatio-temporal representations.