Lune

NeurIPS2024Top-tier venue

Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs

Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Iyer, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, Xichen Pan, Rob Fergus

Abstract

We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach. While stronger language models can enhance multimodal capabilities, the design choices for vision components are often insufficiently explored and disconnected from visual representation learning research. This gap hinders accurate sensory grounding in real-world scenarios. Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations, offering new insights into different models and architectures -- self-supervised, strongly supervised, or combinations thereof -- based on experiments with over 20 vision encoders. We critically examine existing MLLM benchmarks, address the difficulties involved in consolidating and interpreting results from various tasks, and introduce a new vision-centric benchmark, CV-Bench. To further improve visual grounding, we propose the Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that integrates high-resolution vision features with LLMs while reducing the number of tokens. Additionally, we discuss the curation of high-quality visual instruction-tuning data from publicly available sources, emphasizing the importance of data source balancing and distribution ratio. Collectively, Cambrian-1 not only achieves state-of-the-art performance but also serves as a comprehensive, open cookbook for instruction-tuned MLLMs. We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes. We hope our release will inspire and accelerate advancements in multimodal systems and visual representation learning.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

376 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Towards Unified Multimodal Pretraining cites Perception Encoder: The best visual embeddings are not at the output of the network
  • Towards Unified Multimodal Pretraining cites Learning Transferable Visual Models From Natural Language Supervision
  • Segment Anything cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Segment Anything cites Learning Transferable Visual Models From Natural Language Supervision
  • MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Perception Encoder: The best visual embeddings are not at the output of the network cites Segment Anything
  • Perception Encoder: The best visual embeddings are not at the output of the network cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Perception Encoder: The best visual embeddings are not at the output of the network cites Learning Transferable Visual Models From Natural Language Supervision
  • GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  • Learning Transferable Visual Models From Natural Language Supervision cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers376

  1. Perception Encoder: The best visual embeddings are not at the output of the networkDaniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho et al.NeurIPS 2025 · 359 citations
  2. Towards Unified Multimodal PretrainingShengbang Tong, David Fan, John Nguyen, Ellis Brown et al.ICML 2026 · 234 citations
  3. VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech InteractionChaoyou Fu, Haojia Lin, Xiong Wang, Yifan Zhang et al.NeurIPS 2025 · 234 citations
  4. MMSI-Bench: A Benchmark for Multi-Image Spatial IntelligenceSihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang et al.ICLR 2026 · 195 citations
  5. GPG: A Simple and Strong Reinforcement Learning Baseline for Model ReasoningXiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei et al.ICLR 2026 · 168 citations

Builds on67

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  4. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  5. Segment AnythingAlexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao et al.ICCV 2023 · 13,211 citations

Related papers

  1. PGT: Procedurally Generated Tasks for improving visual grounding in MLLMsRim Assouel, Amir Bar, Michal Drozdzal, Adriana Romero-SorianoICML 2026
  2. MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMsYunqiu Xu, Linchao Zhu, Yi YangICCV 2025 · 7 citations
  3. Finer: Investigating and Enhancing Fine-Grained Visual Concept Recognition in Large Vision Language ModelsJeonghwan Kim, Heng JiEMNLP 2024 · 4 citations
  4. VisualAgentBench: Towards Large Multimodal Models as Visual Foundation AgentsXiao Liu, Tianjie Zhang, Yu Gu, Iat Long Iong et al.ICLR 2025
  5. Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction TuningJi Hyeok Jung, Eun Tae Kim, Seo Yeon Kim, Joo Ho Lee et al.CVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines