ACL Findings2022
Comprehensive Multi-Modal Interactions for Referring Image Segmentation
Abstract
We investigate Referring Image Segmentation (RIS), which outputs a segmentation map corresponding to the natural language description. Addressing RIS efficiently requires considering the interactions happening across visual and linguistic modalities and the interactions within each modality. Existing methods are limited because they either compute different forms of interactions sequentially (leading to error propagation) or ignore intramodal interactions. We address this limitation by performing all three interactions simultaneously through a Synchronous Multi-Modal Fusion Module (SFM). Moreover, to produce refined segmentation masks, we propose a novel Hierarchical Cross-Modal Aggregation Module (HCAM), where linguistic features facilitate the exchange of contextual information across the visual hierarchy. We present thorough ablation studies and validate our approach's performance on four benchmark datasets, showing considerable performance gains over the existing state-ofthe-art (SOTA) methods.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
2 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Referring Image Editing: Object-Level Image Editing via Referring Expressions cites Bottom-Up Shift and Reasoning for Referring Image Segmentation
- Referring Image Editing: Object-Level Image Editing via Referring Expressions cites Bi-Directional Relationship Inferring Network for Referring Image Segmentation
- Bottom-Up Shift and Reasoning for Referring Image Segmentation cites See-Through-Text Grouping for Referring Image Segmentation
- Bottom-Up Shift and Reasoning for Referring Image Segmentation cites Bi-Directional Relationship Inferring Network for Referring Image Segmentation
- Bottom-Up Shift and Reasoning for Referring Image Segmentation cites Referring Image Segmentation via Cross-Modal Progressive Comprehension
- Bi-Directional Relationship Inferring Network for Referring Image Segmentation cites See-Through-Text Grouping for Referring Image Segmentation
- Referring Image Segmentation via Cross-Modal Progressive Comprehension cites See-Through-Text Grouping for Referring Image Segmentation
Cited by top-tier papers2
Builds on6
- See-Through-Text Grouping for Referring Image SegmentationDing-Jie Chen, Songhao Jia, Yi-Chen Lo, Hwann-Tzong Chen et al.ICCV 2019 · 153 citations
- Learning Cross-Modal Context Graph for Visual GroundingYongfei Liu, Bo Wan, Xiaodan Zhu, Xuming HeAAAI 2020 · 100 citations
- Bottom-Up Shift and Reasoning for Referring Image SegmentationSibei Yang, Meng Xia, Guanbin Li, Hong-Yu Zhou et al.CVPR 2021
- Bi-Directional Relationship Inferring Network for Referring Image SegmentationZhiwei Hu, Guang Feng, Jiayu Sun, Lihe Zhang et al.CVPR 2020
- Referring Image Segmentation via Cross-Modal Progressive ComprehensionShaofei Huang, Tianrui Hui, Si Liu, Guanbin Li et al.CVPR 2020
Related papers
- CARIS: Context-Aware Referring Image SegmentationSun'ao Liu, Yiheng Zhang, Zhaofan Qiu, Hongtao Xie et al.ACM MM 2023 · 34 citations
- Locate Then Segment: A Strong Pipeline for Referring Image SegmentationYa Jing, Tao Kong, Wei Wang, Liang Wang et al.CVPR 2021
- LQMFormer: Language-Aware Query Mask Transformer for Referring Image SegmentationNisarg A. Shah, Vibashan VS, Vishal M. PatelCVPR 2024 · 11 citations
- Two-stage Visual Cues Enhancement Network for Referring Image SegmentationYang Jiao, Zequn Jie, Weixin Luo, Jingjing Chen et al.ACM MM 2021 · 24 citations
- Encoder Fusion Network With Co-Attention Embedding for Referring Image SegmentationGuang Feng, Zhiwei Hu, Lihe Zhang, Huchuan LuCVPR 2021
