ACL Findings2021
Semantic Relation-aware Difference Representation Learning for Change Captioning
Yunbin Tu, Tingting Yao, Liang Li, Jiedong Lou, Shengxiang Gao, Zhengtao Yu, Chenggang Yan
Abstract
Change captioning is to describe the difference in a pair of images with a natural language sentence. In this task, the distractors, such as the illumination or viewpoint change, bring the huge challenges about learning the difference representation. In this paper, we propose a semantic relation-aware difference representation learning network to explicitly learn the difference representation in the existence of distractors. Specifically, we introduce a selfsemantic relation embedding block to explore the underlying changed objects and design a cross-semantic relation measuring block to localize the real change and learn the discriminative difference representation. Besides, relying on the POS of words, we devise an attentionbased visual switch to dynamically use visual information for caption generation. Extensive experiments show that our method achieves the state-of-the-art performances on CLEVR-Change and Spot-the-Diff datasets 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Self-supervised Cross-view Representation Reconstruction for Change Captioning cites Robust Change Captioning
- Self-supervised Cross-view Representation Reconstruction for Change Captioning cites R3Net: Relation-embedded Representation Reconstruction Network for Change Captioning
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning cites Self-supervised Cross-view Representation Reconstruction for Change Captioning
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning cites Robust Change Captioning
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change Captioning cites R3Net: Relation-embedded Representation Reconstruction Network for Change Captioning
- Image Difference Captioning via Adversarial Preference Optimization cites Self-supervised Cross-view Representation Reconstruction for Change Captioning
- Image Difference Captioning via Adversarial Preference Optimization cites Robust Change Captioning
- Image Difference Captioning via Adversarial Preference Optimization cites R3Net: Relation-embedded Representation Reconstruction Network for Change Captioning
- DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning cites Self-supervised Cross-view Representation Reconstruction for Change Captioning
- DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change Captioning cites Robust Change Captioning
- R3Net: Relation-embedded Representation Reconstruction Network for Change Captioning cites A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
- R3Net: Relation-embedded Representation Reconstruction Network for Change Captioning cites Aligned Dual Channel Graph Convolutional Network for Visual Question Answering
- R3Net: Relation-embedded Representation Reconstruction Network for Change Captioning cites Robust Change Captioning
Cited by top-tier papers7
- Self-supervised Cross-view Representation Reconstruction for Change CaptioningYunbin Tu, Liang Li, Li Su, Zheng-Jun Zha et al.ICCV 2023 · 45 citations
- R3Net: Relation-embedded Representation Reconstruction Network for Change CaptioningYunbin Tu, Liang Li, Chenggang Yan, Shengxiang Gao et al.EMNLP 2021 · 21 citations
- Region-aware Difference Distilling with Attribute-guided Contrastive Regularization for Change CaptioningRong Li, Liang Li, Jiehua Zhang, Qiang Zhao et al.AAAI 2025 · 4 citations
- Image Difference Captioning via Adversarial Preference OptimizationZihan Huang, Junda Wu, Rohan Surana, Tong Yu et al.EMNLP 2025 · 3 citations
- DECIDER: Difference-aware Contrastive Diffusion Model with Adversarial Perturbations for Image Change CaptioningGuojin Zhong, Jinhong Hu, Jiajun Chen, Jin Yuan et al.AAAI 2025 · 3 citations
Builds on7
- Robust Change CaptioningDong Huk Park, Trevor Darrell, Anna RohrbachICCV 2019 · 217 citations
- Controllable Video Captioning With POS Sequence Guidance Based on Gated Fusion NetworkBairui Wang, Lin Ma, Wei Zhang, Wenhao Jiang et al.ICCV 2019 · 183 citations
- A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine TranslationYongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou et al.ACL 2020 · 145 citations
- Adaptive Reconstruction Network for Weakly Supervised Referring Expression GroundingXuejing Liu, Liang Li, Shuhui Wang, Zheng-Jun Zha et al.ICCV 2019 · 93 citations
- Aligned Dual Channel Graph Convolutional Network for Visual Question AnsweringQingbao Huang, Jielong Wei, Yi Cai, Changmeng Zheng et al.ACL 2020 · 79 citations
Related papers
- Change Entity-guided Heterogeneous Representation Disentangling for Change CaptioningYi Li, Yunbin Tu, Liang Li, Li Su et al.ACL Findings 2025 · 2 citations
- Scene Graph with 3D Information for Change CaptioningZeming Liao, Qingbao Huang, Yu Liang, Mingyi Fu et al.ACM MM 2021 · 17 citations
- Context-aware Difference Distilling for Multi-change CaptioningYunbin Tu, Liang Li, Li Su, Zheng-Jun Zha et al.ACL 2024
- Revisiting Change Captioning from Self-supervised Global-Part AlignmentFeixiao Lv, Rui Wang, Lihua JingAAAI 2025 · 1 citation
- Image Difference Captioning with Pre-training and Contrastive LearningLinli Yao, Weiying Wang, Qin JinAAAI 2022 · 66 citations
