Lune

Abstract

Multi-modal large language models (MLLMs) have shown incredible capabilities in a variety of 2D vision and language tasks. We extend MLLMs' perceptual capabilities to ground and reason about images in 3-dimensional space. To that end, we first develop a largescale pretraining dataset for 2D and 3D called LV3D by combining multiple existing 2D and 3D recognition datasets under a common task formulation: as multi-turn question-answering. Next, we introduce a new MLLM named Cube-LLM and pre-train it on LV3D. We show that pure data scaling makes a strong 3D perception capability without 3D specific architectural design or training objective. Cube-LLM exhibits intriguing properties similar to LLMs: (1) Cube-LLM can apply chainof-thought prompting to improve 3D understanding from 2D context information. (2) Cube-LLM can follow complex and diverse instructions and adapt to versatile input and output formats. (3) Cube-LLM can be visually prompted such as 2D box or a set of candidate 3D boxes from specialists. Our experiments on outdoor benchmarks demonstrate that Cube-LLM significantly outperforms existing baselines by 21.3 points of APBEV on the Talk2Car dataset for 3D grounded reasoning and 17.7 points on the DriveLM dataset for complex reasoning about driving scenarios, respectively. Cube-LLM also shows competitive results in general MLLM benchmarks such as refCOCO for 2D grounding with (87.0) average score, as well as visual question answering benchmarks such as VQAv2, GQA, SQA, POPE, etc. for complex reasoning. Our project is available at https://janghyuncho.github.io/Cube-LLM .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

11 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D Editing cites Learning Transferable Visual Models From Natural Language Supervision
  • Segment Anything cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Segment Anything cites Learning Transferable Visual Models From Natural Language Supervision
  • PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding cites Segment Anything
  • PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding cites Visual Instruction Tuning
  • PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding cites Learning Transferable Visual Models From Natural Language Supervision
  • Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • RefAV: Towards Planning-Centric Scenario Mining cites Segment Anything
  • RefAV: Towards Planning-Centric Scenario Mining cites Visual Instruction Tuning
  • RefAV: Towards Planning-Centric Scenario Mining cites Learning Transferable Visual Models From Natural Language Supervision
  • FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving cites Learning Transferable Visual Models From Natural Language Supervision
  • FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers11

  1. FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous DrivingShuang Zeng, Xinyuan Chang, Mengwei Xie, Xinran Liu et al.NeurIPS 2025 · 228 citations
  2. PerceptionLM: Open-Access Data and Models for Detailed Visual UnderstandingJang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi, Triantafyllos Afouras et al.NeurIPS 2025 · 97 citations
  3. MVInpainter: Learning Multi-View Consistent Inpainting to Bridge 2D and 3D EditingChenjie Cao, Chaohui Yu, Fan Wang, Xiangyang Xue et al.NeurIPS 2024 · 36 citations
  4. DriveDreamer-2: LLM-Enhanced World Models for Diverse Driving Video GenerationGuosheng Zhao, Xiaofeng Wang, Zheng Zhu, Xinze Chen et al.AAAI 2025 · 31 citations
  5. RefAV: Towards Planning-Centric Scenario MiningCainan Davidson, Deva Ramanan, Neehar PeriCVPR 2026 · 17 citations

Builds on29

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  3. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  4. Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations
  5. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations

Related papers

  1. MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMsErik A. Daxberger, Nina Wenzel, David Griffiths, Haiming Gang et al.ICCV 2025 · 10 citations
  2. LLaVA³: Representing 3D Scenes Like a Cubist Painter to Boost 3D Scene Understanding of VLMsDoriand Petit, Steve Bourgeois, Vincent Gay-Bellile, Florian Chabot et al.AAAI 2026
  3. 3D-LLM: Injecting the 3D World into Large Language ModelsYining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng et al.NeurIPS 2023 · 662 citations
  4. Vid-LLM: A Compact Video-based 3D Multimodal LLM with Reconstruction-Reasoning SynergyHaijier Chen, Bo Xu, Shoujian Zhang, Haoze Liu et al.ICLR 2026 · 6 citations
  5. Struct2D: A Perception-Guided Framework for Spatial Reasoning in MLLMsFangrui Zhu, Hanhui Wang, Yiming Xie, Jing Gu et al.NeurIPS 2025 · 7 citations
Dusk over the sea between two cliffs drawn in fine vertical lines