Lune

NeurIPS2025Top-tier venue

3DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding

Xiaohu Huang, Jingjing Wu, Qunyi Xie, Kai Han

Abstract

Recent advances in scene understanding have leveraged multimodal large language models (MLLMs) for 3D reasoning by capitalizing on their strong 2D pretraining. However, the lack of explicit 3D data during MLLM pretraining limits 3D representation capability. In this paper, we investigate the 3D-awareness of MLLMs by evaluating multi-view correspondence and reveal a strong positive correlation between the quality of 3D-aware representation and downstream task performance. Motivated by this, we propose 3DRS, a framework that enhances MLLM 3D Representation learning by introducing Supervision from pretrained 3D foundation models. Our approach aligns MLLM visual features with rich 3D knowledge distilled from 3D models, effectively improving scene understanding. Extensive experiments across multiple benchmarks and MLLMs-including visual grounding, captioning, and question answering-demonstrate consistent performance gains.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LARA: Latent Action Representation Alignment for Vision-Language-Action Models cites An Embodied Generalist Agent in 3D World
  • 3D-LLM: Injecting the 3D World into Large Language Models cites Visual Instruction Tuning
  • 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment cites 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
  • An Embodied Generalist Agent in 3D World cites 3DVG-Transformer: Relation Modeling for Visual Grounding on Point Clouds
  • An Embodied Generalist Agent in 3D World cites 3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment
  • An Embodied Generalist Agent in 3D World cites Visual Instruction Tuning
  • Vision-aligned Latent Reasoning for Multi-modal Large Language Model cites Visual Instruction Tuning
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Thinking with Geometry: Active Geometry Integration for Spatial ReasoningHaoyuan Li, Qihang Cao, Tao Tang, Kun Xiang et al.ICML 2026 · 12 citations
  2. Vision-aligned Latent Reasoning for Multi-modal Large Language ModelByungwoo Jeon, Yoonwoo Jeong, Hyunseok Lee, Minsu Cho et al.ICML 2026 · 7 citations
  3. LARA: Latent Action Representation Alignment for Vision-Language-Action ModelsMengya Liu, Baoxiong Jia, Jiangyong Huang, Jingze Zhang et al.ICML 2026 · 3 citations

Builds on40

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. 3D-LLM: Injecting the 3D World into Large Language ModelsYining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng et al.NeurIPS 2023 · 662 citations
  3. An Embodied Generalist Agent in 3D WorldJiangyong Huang, Silong Yong, Xiaojian Ma, Xiongkun Linghu et al.ICML 2024 · 361 citations
  4. 3D-VisTA: Pre-trained Transformer for 3D Vision and Text AlignmentZiyu Zhu, Xiaojian Ma, Yixin Chen, Zhidong Deng et al.ICCV 2023 · 247 citations
  5. 3DVG-Transformer: Relation Modeling for Visual Grounding on Point CloudsLichen Zhao, Daigang Cai, Lu Sheng, Dong XuICCV 2021 · 234 citations

Related papers

  1. Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning SynergyHaijier Chen, Bo Xu, Shoujian Zhang, Haoze Liu et al.ICLR 2026 · 6 citations
  2. Language-Image Models with 3D UnderstandingJang Hyun Cho, Boris Ivanovic, Yulong Cao, Edward Schmerling et al.ICLR 2025 · 2 citations
  3. 3D-Aware Vision-Language Models Fine-Tuning with Geometric DistillationSeonho Lee, Jiho Choi, Inha Kang, Jiwook Kim et al.EMNLP Findings 2025 · 2 citations
  4. Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry PriorsDuo Zheng, Shijia Huang, Yanyang Li, Liwei WangNeurIPS 2025 · 130 citations
  5. S^2-MLLM: Boosting Spatial Reasoning Capability of MLLMs for 3D Visual Grounding with Structural GuidanceBeining Xu, Siting Zhu, Zhao Jin, Junxian Li et al.CVPR 2026
Dusk over the sea between two cliffs drawn in fine vertical lines