Lune

ACM MM2020Top-tier venue

Language-Aware Fine-Grained Object Representation for Referring Expression Comprehension

Heqian Qiu, Hongliang Li, Qingbo Wu, Fanman Meng, Hengcan Shi, Taijin Zhao, King Ngi Ngan

Abstract

Referring expression comprehension expects to accurately locate an object described by a language expression, which requires precise language-aware visual object representations. However, existing methods usually use rectangular object representations, such as object proposal regions and grid regions. They ignore some fine-grained object information like shapes and poses, which are often described in language expressions and important to localize objects. Additionally, rectangular object regions usually contain background contents and irrelevant foreground features, which also decrease the localization performance. To address these problems, we propose a language-aware deformable convolution model (LDC) to learn language-aware fine-grained object representations. Rather than extracting rectangular object representations, LDC adaptively samples a set of key points based on the image and language to represent objects. This type of object representations can capture more fine-grained object information (e.g., shapes and poses) and suppress noises in accordance with language and thus, boosts the object localization performance. Based on the language-aware fine-grained object representation, we next design a bidirectional interaction model (BIM) that leverages a modified co-attention mechanism to build cross-modal bidirectional interactions to further improve the language and object representations. Furthermore, we propose a hierarchical fine-grained representation network (HFRN) to learn language-aware fine-grained object representations and cross-modal bidirectional interactions at local word level and global sentence level, respectively. Our proposed method outperforms the state-of-the-art methods on the RefCOCO, RefCOCO+ and RefCOCOg datasets.

Ask Lune

Know where this paper stands in minutes, not an afternoon.

Lune can read the top-tier papers that cite it and the ones closest to it, and tells you what it claims, what came after and where it falls short, each point cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

its abstract · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Visual Grounding with Multi-modal Conditional Adaptation cites Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual Grounding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Shifting More Attention to Visual Backbone: Query-modulated Refinement Networks for End-to-End Visual GroundingJiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang et al.CVPR 2022 · 89 citations
  2. ProposalCLIP: Unsupervised Open-Category Object Proposal Generation via Exploiting CLIP CuesHengcan Shi, Munawar Hayat, Yicheng Wu, Jianfei CaiCVPR 2022 · 59 citations
  3. Open-Vocabulary Object Detection via Scene Graph DiscoveryHengcan Shi, Munawar Hayat, Jianfei CaiACM MM 2023 · 20 citations
  4. Visual Grounding with Multi-modal Conditional AdaptationRuilin Yao, Shengwu Xiong, Yichen Zhao, Yi RongACM MM 2024 · 19 citations
  5. RefCrowd: Grounding the Target in Crowd with Referring ExpressionsHeqian Qiu, Hongliang Li, Taijin Zhao, Lanxiao Wang et al.ACM MM 2022 · 10 citations

Related papers

  1. Locate Then Segment: A Strong Pipeline for Referring Image SegmentationYa Jing, Tao Kong, Wei Wang, Liang Wang et al.CVPR 2021
  2. Bottom-Up and Bidirectional Alignment for Referring Expression ComprehensionLiuwu Li, Yuqi Bu, Yi CaiACM MM 2021 · 11 citations
  3. Exploring Logical Reasoning for Referring Expression ComprehensionYing Cheng, Ruize Wang, Jiashuo Yu, Rui-Wei Zhao et al.ACM MM 2021 · 12 citations
  4. Leveraging Debiased Cross-Modal Attention Maps and Code-Based Reasoning for Zero-Shot Referring Expression ComprehensionJuntao Chen, Wen Shen, Zhihua Wei, Lijun Sun et al.ICCV 2025 · 1 citation
  5. Revisiting Counterfactual Problems in Referring Expression ComprehensionZhihan Yu, Ruifan LiCVPR 2024 · 6 citations
Dusk over the sea between two cliffs drawn in fine vertical lines