Lune

EMNLP Findings2023

CCIM: Cross-modal Cross-lingual Interactive Image Translation

Cong Ma, Yaping Zhang, Mei Tu, Yang Zhao, Yu Zhou, Chengqing Zong

Abstract

Text image machine translation (TIMT) which translates source language text images into target language texts has attracted intensive attention in recent years. Although the endto-end TIMT model directly generates target translation from encoded text image features with an efficient architecture, it lacks the recognized source language information resulting in a decrease in translation performance. In this paper, we propose a novel Cross-modal Cross-lingual Interactive Model (CCIM) to incorporate source language information by synchronously generating source language and target language results through an interactive attention mechanism between two language decoders. Extensive experimental results have shown the interactive decoder significantly outperforms end-to-end TIMT models and has faster decoding speed with smaller model size than cascade models. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Towards Fully Automated Manga Translation cites What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model Analysis
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on5

  1. What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model AnalysisJeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park et al.ICCV 2019 · 551 citations
  2. Synchronous Speech Recognition and Speech-to-Text Translation with Interactive DecodingYuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou et al.AAAI 2020 · 73 citations
  3. Towards Fully Automated Manga TranslationRyota Hinami, Shonosuke Ishiwatari, Kazuhiko Yasuda, Yusuke MatsuiAAAI 2021 · 41 citations
  4. Synchronous Interactive Decoding for Multilingual Neural Machine TranslationHao He, Qian Wang, Zhipeng Yu, Yang Zhao et al.AAAI 2021 · 13 citations
  5. Improving Multilingual Neural Machine Translation with Auxiliary Source LanguagesWeijia Xu, Yuwei Yin, Shuming Ma, Dongdong Zhang et al.EMNLP Findings 2021 · 6 citations

Related papers

  1. In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence ModelYanzhi Tian, Xiang Li, Zeming Liu, Yuhang Guo et al.EMNLP Findings 2023 · 3 citations
  2. Exploring Better Text Image Translation with Multimodal CodebookZhibin Lan, Jiawei Yu, Xiang Li, Wen Zhang et al.ACL 2023 · 12 citations
  3. PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image TranslationShaolin Zhu, Shangjie Li, Yikun Lei, Deyi XiongACL 2023 · 12 citations
  4. VQA-Augmented Machine Translation with Cross-Modal Contrastive LearningZhihui Zhang, Shiliang Sun, Jing Zhao, Tengfei Song et al.EMNLP Findings 2025
  5. InImageTrans: Multimodal LLM-based Text Image Machine TranslationFei Zuo, Kehai Chen, Yu Zhang, Zhengshan Xue et al.ACL Findings 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines