Lune

ACM MM2020Top-tier venue

Dynamic Context-guided Capsule Network for Multimodal Machine Translation

Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, Jiebo Luo

Abstract

Multimodal machine translation (MMT), which mainly focuses on enhancing text-only translation with visual features, has attracted considerable attention from both computer vision and natural language processing communities. Most current MMT models resort to attention mechanism, global context modeling or multimodal joint representation learning to utilize visual features. However, the attention mechanism lacks sufficient semantic interactions between modalities while the other two provide fixed visual context, which is unsuitable for modeling the observed variability when generating translation. To address the above issues, in this paper, we propose a novel Dynamic Context-guided Capsule Network (DCCN) for MMT. Specifically, at each timestep of decoding, we first employ the conventional source-target attention to produce a timestep-specific source-side context vector. Next, DCCN takes this vector as input and uses it to guide the iterative extraction of related visual features via a context-guided dynamic routing mechanism. Particularly, we represent the input image with global and regional visual features, we introduce two parallel DCCNs to model multimodal context vectors with visual features at different granularities. Finally, we obtain two multimodal context vectors, which are fused and incorporated into the decoder for the prediction of the target word. Experimental results on the Multi30K dataset of English-to-German and English-to-French translation demonstrate the superiority of DCCN. Our code is available on https://github.com/DeepLearnXMU/MM-DCCN.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

21 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation cites VALHALLA: Visual Hallucination for Machine Translation
  • CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation cites A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
  • MSCTD: A Multimodal Sentiment Chat Translation Dataset cites A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
  • VALHALLA: Visual Hallucination for Machine Translation cites A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
  • Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models cites A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
  • Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models cites On Vision Features in Multimodal Machine Translation
  • On Vision Features in Multimodal Machine Translation cites A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers21

  1. On Vision Features in Multimodal Machine TranslationBei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou et al.ACL 2022 · 82 citations
  2. Multi-Modal Latent Space Learning for Chain-of-Thought Reasoning in Language ModelsLiqi He, Zuchao Li, Xiantao Cai, Ping WangAAAI 2024 · 38 citations
  3. VALHALLA: Visual Hallucination for Machine TranslationYi Li, Rameswar Panda, Yoon Kim, Chun-Fu Richard Chen et al.CVPR 2022 · 31 citations
  4. CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine TranslationDevaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li et al.ICCV 2023 · 28 citations
  5. MSCTD: A Multimodal Sentiment Chat Translation DatasetYunlong Liang, Fandong Meng, Jinan Xu, Yufeng Chen et al.ACL 2022 · 27 citations

Builds on2

  1. A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine TranslationYongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou et al.ACL 2020 · 145 citations
  2. Dual Directed Capsule Network for Very Low Resolution Image RecognitionManeet Singh, Shruti Nagpal, Richa Singh, Mayank VatsaICCV 2019 · 56 citations

Related papers

  1. VQA-Augmented Machine Translation with Cross-Modal Contrastive LearningZhihui Zhang, Shiliang Sun, Jing Zhao, Tengfei Song et al.EMNLP Findings 2025
  2. Multimodal Machine Translation with Text-Image In-depth QuestioningYue Gao, Jing Zhao, Shiliang Sun, Xiaosong Qiao et al.ACL Findings 2025
  3. Neural Machine Translation with Phrase-Level Universal Visual RepresentationsQingkai Fang, Yang FengACL 2022
  4. Efficient Object-Level Visual Context Modeling for Multimodal Machine Translation: Masking Irrelevant Objects Helps GroundingDexin Wang, Deyi XiongAAAI 2021 · 45 citations
  5. Visual Agreement Regularized Training for Multi-Modal Machine TranslationPengcheng Yang, Boxing Chen, Pei Zhang, Xu SunAAAI 2020 · 34 citations
Dusk over the sea between two cliffs drawn in fine vertical lines