3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
Xiaohu Huang, Jingjing Wu, Qunyi Xie, Kai Han
摘要
Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D Representation learning by introducing Supervision from pretrained 3D foundation models. Our approach aligns MLLM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive experiments across multiple benchmarks and MLLMs-including visual grounding, captioning, and question answering-demonstrate consistent performance gains.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 3 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- LARA: Latent Action Representation Alignment for Vision-Language-Action Models 引用了 An Embodied Generalist Agent in 3D World
- 3D-LLM: Injecting the 3D World into Large Language Models 引用了 Visual Instruction Tuning
- 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment 引用了 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
- An Embodied Generalist Agent in 3D World 引用了 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
- An Embodied Generalist Agent in 3D World 引用了 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
- An Embodied Generalist Agent in 3D World 引用了 Visual Instruction Tuning
- Vision-aligned Latent Reasoning for Multi-modal Large Language Model 引用了 Visual Instruction Tuning
引用它的顶会 Paper3
- Thinking with Geometry: Active Geometry Integration for Spatial ReasoningHaoyuan Li, Qihang Cao, Tao Tang, Kun Xiang 等ICML 2026 · 12 次引用
- Vision-aligned Latent Reasoning for Multi-modal Large Language ModelByungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho 等ICML 2026 · 7 次引用
- LARA: Latent Action Representation Alignment for Vision-Language-Action ModelsMengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang 等ICML 2026 · 3 次引用
它引用的 Paper40
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 次引用
- 3D-LLM: Injecting the 3D World into Large Language ModelsYining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng 等NeurIPS 2023 · 662 次引用
- An Embodied Generalist Agent in 3D WorldJiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu 等ICML 2024 · 361 次引用
- 3D-VisTA: Pre-trained Transformer for 3D Vision and Text AlignmentZiyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng 等ICCV 2023 · 247 次引用
- 3DVG-Transformer: Relation Modeling for Visual Grounding on Point CloudsLichen Zhao, Daigang Cai, Lu Sheng, Dong XuICCV 2021 · 234 次引用
相关 Paper
- Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning SynergyHaijier Chen, Bo Xu, Shoujian Zhang, Haoze Liu 等ICLR 2026 · 6 次引用
- Language-Image Models with 3D UnderstandingJang Hyun Cho, Boris Ivanovic, Yulong Cao, Edward Schmerling 等ICLR 2025 · 2 次引用
- 3D-Aware Vision-Language Models Fine-Tuning with Geometric DistillationSeonho Lee, Jiho Choi, Inha Kang, Jiwook Kim 等EMNLP Findings 2025 · 2 次引用
- Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry PriorsDuo Zheng, Shijia Huang, Yanyang Li, Liwei WangNeurIPS 2025 · 130 次引用
- S^2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural GuidanceBeining Xu, Siting Zhu, Zhao Jin, Junxian Li 等CVPR 2026
