Lune

EMNLP Findings2023

Scene Graph Enhanced Pseudo-Labeling for Referring Expression Comprehension

Cantao Wu, Yi Cai, Liuwu Li, Jiexin Wang

Abstract

Referring Expression Comprehension (ReC) is a task that involves localizing objects in images based on natural language expressions. Most ReC methods typically approach the task as a supervised learning problem. However, the need for costly annotations, such as clear image-text pairs or region-text pairs, hinders the scalability of existing approaches. In this work, we propose a novel scene graph-based framework that automatically generates highquality pseudo region-query pairs. Our method harnesses scene graphs to capture the relationships between objects in images and generate expressions enriched with relation information.

To ensure accurate mapping between visual regions and text, we introduce an external module that employs a calibration algorithm to filter out ambiguous queries. Additionally, we employ a rewriter module to enhance the diversity of our generated pseudo queries through rewriting. Extensive experiments demonstrate that our method outperforms previous pseudolabeling methods by about 10%, 12%, and 11% on RefCOCO, RefCOCO+, and RefCOCOg, respectively. Furthermore, it surpasses the stateof-the-art unsupervised approach by more than 15% on the RefCOCO dataset.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension cites TransVG: End-to-End Visual Grounding with Transformers
  • MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension cites Learning Transferable Visual Models From Natural Language Supervision
  • MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension cites Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning
  • Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual Prompting cites Learning Transferable Visual Models From Natural Language Supervision
  • ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension cites Adaptive Reconstruction Network for Weakly Supervised Referring Expression Grounding
  • ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension cites Learning Transferable Visual Models From Natural Language Supervision
  • Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning cites TransVG: End-to-End Visual Grounding with Transformers
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression ComprehensionTing Liu, Zunnan Xu, Yue Hu, Liangtao Shi et al.EMNLP 2024 · 6 citations
  2. Graph-of-Mark: Promote Spatial Reasoning in Multimodal Language Models with Graph-Based Visual PromptingGiacomo Frisoni, Lorenzo Molfetta, Mattia Buzzoni, Gianluca MoroAAAI 2026

Builds on10

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. TransVG: End-to-End Visual Grounding with TransformersJiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou et al.ICCV 2021 · 468 citations
  3. ReCLIP: A Strong Zero-Shot Baseline for Referring Expression ComprehensionSanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner et al.ACL 2022 · 172 citations
  4. Improving Visual Grounding with Visual-Linguistic Verification and Iterative ReasoningLi Yang, Yan Xu, Chunfeng Yuan, Wei Liu et al.CVPR 2022 · 146 citations
  5. Adaptive Reconstruction Network for Weakly Supervised Referring Expression GroundingXuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha et al.ICCV 2019 · 93 citations

Related papers

  1. Whether you can locate or not? Interactive Referring Expression GenerationFulong Ye, Yuxing Long, Fangxiang Feng, Xiaojie WangACM MM 2023 · 6 citations
  2. Towards Further Comprehension on Referring Expression with RationaleRengang Li, Baoyu Fan, Xiaochuan Li, Runze Zhang et al.ACM MM 2022 · 2 citations
  3. RefTeacher: A Strong Baseline for Semi-Supervised Referring Expression ComprehensionJiamu Sun, Gen Luo, Yiyi Zhou, Xiaoshuai Sun et al.CVPR 2023
  4. Graph-Structured Referring Expression Reasoning in the WildSibei Yang, Guanbin Li, Yizhou YuCVPR 2020
  5. RefCLIP: A Universal Teacher for Weakly Supervised Referring Expression ComprehensionLei Jin, Gen Luo, Yiyi Zhou, Xiaoshuai Sun et al.CVPR 2023
Dusk over the sea between two cliffs drawn in fine vertical lines