Lune

NeurIPS2024Top-tier venue

Referencing Where to Focus: Improving Visual Grounding with Referential Query

Yabing Wang, Zhuotao Tian, Qingpei Guo, Zheng Qin, Sanping Zhou, Ming Yang, Le Wang

Abstract

Visual Grounding aims to localize the referring object in an image given a natural language expression. Recent advancements in DETR-based visual grounding methods have attracted considerable attention, as they directly predict the coordinates of the target object without relying on additional efforts, such as pre-generated proposal candidates or pre-defined anchor boxes. However, existing research primarily focuses on designing stronger multi-modal decoder, which typically generates learnable queries by random initialization or by using linguistic embeddings. This vanilla query generation approach inevitably increases the learning difficulty for the model, as it does not involve any target-related information at the beginning of decoding. Furthermore, they only use the deepest image feature during the query learning process, overlooking the importance of features from other levels. To address these issues, we propose a novel approach, called RefFormer. It consists of the query adaption module that can be seamlessly integrated into CLIP and generate the referential query to provide the prior context for decoder, along with a task-specific decoder. By incorporating the referential query into the decoder, we can effectively mitigate the learning difficulty of the decoder, and accurately concentrate on the target object. Additionally, our proposed query adaption module can also act as an adapter, preserving the rich knowledge within CLIP without the need to tune the parameters of the backbone network. Extensive experiments demonstrate the effectiveness and efficiency of our proposed method, outperforming state-of-the-art approaches on five visual grounding benchmarks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • DN-DETR: Accelerate DETR Training by Introducing Query DeNoising cites DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR
  • Spatial Matters: Position-Guided 3D Referring Expression Segmentation cites Versatile Multimodal Controls for Expressive Talking Human Animation
  • Spatial Matters: Position-Guided 3D Referring Expression Segmentation cites HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMs
  • WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and Segmentation cites MDETR - Modulated Detection for End-to-End Multi-Modal Understanding
  • MDETR - Modulated Detection for End-to-End Multi-Modal Understanding cites Learning Transferable Visual Models From Natural Language Supervision
  • Versatile Multimodal Controls for Expressive Talking Human Animation cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. Versatile Multimodal Controls for Expressive Talking Human AnimationZheng Qin, Ruobing Zheng, Yabing Wang, Tianqi Li et al.ACM MM 2025 · 2 citations
  2. HumanSense: From Multimodal Perception to Empathetic Context-Aware Responses Through Reasoning MLLMsZheng Qin, Ruobing Zheng, Yabing Wang, Tianqi Li et al.AAAI 2026 · 2 citations
  3. Spatial Matters: Position-Guided 3D Referring Expression SegmentationYabing Wang, Zhuotao Tian, Le Wang, Zheng Qin et al.CVPR 2026
  4. WeakMCN: Multi-task Collaborative Network for Weakly Supervised Referring Expression Comprehension and SegmentationSilin Cheng, Yang Liu, Xinwei He, Sébastien Ourselin et al.CVPR 2025

Builds on29

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Per-Pixel Classification is Not All You Need for Semantic SegmentationBowen Cheng, Alexander G. Schwing, Alexander KirillovNeurIPS 2021 · 2,196 citations
  3. DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETRShilong Liu, Feng Li, Hao Zhang, Xiao Yang et al.ICLR 2022 · 1,218 citations
  4. MDETR - Modulated Detection for End-to-End Multi-Modal UnderstandingAishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve et al.ICCV 2021 · 1,114 citations
  5. DN-DETR: Accelerate DETR Training by Introducing Query DeNoisingFeng Li, Hao Zhang, Shilong Liu, Jian Guo et al.CVPR 2022 · 879 citations

Related papers

  1. Look Around Before Locating: Considering Content and Structure Information for Visual GroundingShiyi Zheng, Peizhi Zhao, Zhilong Zheng, Peihang He et al.AAAI 2025 · 3 citations
  2. Improving Visual Grounding with Visual-Linguistic Verification and Iterative ReasoningLi Yang, Yan Xu, Chunfeng Yuan, Wei Liu et al.CVPR 2022 · 146 citations
  3. Referring Transformer: A One-step Approach to Multi-task Visual GroundingMuchen Li, Leonid SigalNeurIPS 2021 · 270 citations
  4. RefDetector: A Simple Yet Effective Matching-based Method for Referring Expression ComprehensionYabing Wang, Zhuotao Tian, Zheng Qin, Sanping Zhou et al.AAAI 2025 · 2 citations
  5. Task-aware Cross-modal Feature Refinement Transformer with Large Language Models for Visual GroundingWenbo Chen, Zhen Xu, Ruotao Xu, Si Wu et al.CVPR 2025
Dusk over the sea between two cliffs drawn in fine vertical lines