3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding

Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a transformative paradigm for enhancing the reasoning capabilities of Large Language Models (LLMs), yet its potential in 3D scene understanding remains underexplored. Existing approaches largely rely on Supervised Fine-Tuning (SFT), where the tokenlevel cross-entropy loss acts as an indirect proxy for optimization, leading to a misalignment between training objectives and task performances. To bridge this gap, we present 3D-RFT, a systematic framework to extend RLVR to video-based 3D perception and reasoning. 3D-RFT shifts the paradigm by directly optimizing the model towards evaluation metrics, performing reinforcement fine-tuning using Group Relative Policy Optimization (GRPO) with strictly verifiable reward functions, which directly stem from metrics like 3D IoU and F1-Score to provide more effective learning signals. Extensive experiments demonstrate that 3D-RFT-4B achieves state-of-the-art performance on various video-based 3D scene understanding tasks. Notably, 3D-RFT-4B significantly outperforms larger models (e.g., VG LLM-8B) on 3D video detection, 3D visual grounding, and spatial reasoning benchmarks. We further showcase advantages of 3D-RFT, such as robust efficacy, and provide valuable insights into training strategies and data impact. We hope 3D-RFT can serve as a robust and promising paradigm for future development of 3D scene understanding. Code is available on project page.