Lune

CVPR Findings2026

UGLMM: Towards Unified Vision Grounding with Large Multimodal Model

Xiangheng Shan, Li Zhou, Zenghui Sun, Shichao Dong, Nong Sang, Jinsong Lan, Xiaoyong Zhu, Bo Zheng, Changxin Gao, Kaifu Zhang

Abstract

Despite the promising advances that current Large Multimodal Models (LMMs) have made in various vision tasks, unifying and achieving good performance across all vision grounding tasks remains a major challenge. We identify two key reasons for this: sub-optimal grounding paradigms and neglect of the unity between mask-requiring and boxrequiring grounding tasks. To address the above issues, we propose UGLMM, the first generalist LMM that unifies and achieves promising performance across all the vision grounding tasks including language-referred object grounding, category-based vision grounding, visual-prompt-based segmentation, and grounded conversation tasks. To unify all these grounding tasks, we introduce a unified LMM grounding paradigm (Uni-Ground paradigm), which flexibly employs the most suitable grounding process for different vision grounding tasks, enabling UGLMM to handle these tasks effectively with one model. UGLMM also highlights a mask-box grounding co-training strategy, which helps our model unify and perform better on both box-requiring and mask-requiring vision grounding tasks. Additionally, we introduce a grounding query selector and leverage multiple [GND] tokens to further enhance the vision grounding capability of UGLMM. Extensive experiments demonstrate that our UGLMM achieves state-of-the-art performance across a variety of tasks with one unified LMM. We

believe our UGLMM makes a promising advance in developing a strong generalist LMM. Our code and trained models will be here: https://github.com/slonetime/UGLMM.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning cites MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models
  • Visual Instruction Tuning cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on36

  1. Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
  2. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  3. Deformable DETR: Deformable Transformers for End-to-End Object DetectionXizhou Zhu, Weijie Su, Lewei Lu, Bin Li et al.ICLR 2021 · 7,353 citations
  4. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction TuningWenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong et al.NeurIPS 2023 · 4,013 citations
  5. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language ModelsDeyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li et al.ICLR 2024 · 3,079 citations

Related papers

  1. GLaMM: Pixel Grounding Large Multimodal ModelHanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman M. Shaker et al.CVPR 2024 · 113 citations
  2. Hugging Visual Prompt and Segmentation Tokens: Consistency Learning for Fine-Grained Visual Understanding in MLLMsjing yang, Sen Yang, Boqiang Duan, Ming Dai et al.CVPR 2026
  3. GroundingGPT: Language Enhanced Multi-modal Grounding ModelZhaowei Li, Qi Xu, Dong Zhang, Hang Song et al.ACL 2024 · 29 citations
  4. GeM-VG: Towards Generalized Multi-image Visual Grounding with Multimodal Large Language ModelsShurong Zheng, Yousong Zhu, Hongyin Zhao, Fan Yang et al.AAAI 2026
  5. X-SAM: From Segment Anything to Any SegmentationHao Wang, Limeng Qiao, Zequn Jie, Zhijian Huang et al.AAAI 2026 · 16 citations
Dusk over the sea between two cliffs drawn in fine vertical lines