EMNLP Findings2023
CCIM: Cross-modal Cross-lingual Interactive Image Translation
Cong Ma, Yaping Zhang, Mei Tu, Yang Zhao, Yu Zhou, Chengqing Zong
Abstract
Text image machine translation (TIMT) which translates source language text images into target language texts has attracted intensive attention in recent years. Although the endto-end TIMT model directly generates target translation from encoded text image features with an efficient architecture, it lacks the recognized source language information resulting in a decrease in translation performance. In this paper, we propose a novel Cross-modal Cross-lingual Interactive Model (CCIM) to incorporate source language information by synchronously generating source language and target language results through an interactive attention mechanism between two language decoders. Extensive experimental results have shown the interactive decoder significantly outperforms end-to-end TIMT models and has faster decoding speed with smaller model size than cascade models. 1
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
Builds on5
- What Is Wrong With Scene Text Recognition Model Comparisons? Dataset and Model AnalysisJeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park et al.ICCV 2019 · 551 citations
- Synchronous Speech Recognition and Speech-to-Text Translation with Interactive DecodingYuchen Liu, Jiajun Zhang, Hao Xiong, Long Zhou et al.AAAI 2020 · 73 citations
- Towards Fully Automated Manga TranslationRyota Hinami, Shonosuke Ishiwatari, Kazuhiko Yasuda, Yusuke MatsuiAAAI 2021 · 41 citations
- Synchronous Interactive Decoding for Multilingual Neural Machine TranslationHao He, Qian Wang, Zhipeng Yu, Yang Zhao et al.AAAI 2021 · 13 citations
- Improving Multilingual Neural Machine Translation with Auxiliary Source LanguagesWeijia Xu, Yuwei Yin, Shuming Ma, Dongdong Zhang et al.EMNLP Findings 2021 · 6 citations
Related papers
- In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence ModelYanzhi Tian, Xiang Li, Zeming Liu, Yuhang Guo et al.EMNLP Findings 2023 · 3 citations
- Exploring Better Text Image Translation with Multimodal CodebookZhibin Lan, Jiawei Yu, Xiang Li, Wen Zhang et al.ACL 2023 · 12 citations
- PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image TranslationShaolin Zhu, Shangjie Li, Yikun Lei, Deyi XiongACL 2023 · 12 citations
- VQA-Augmented Machine Translation with Cross-Modal Contrastive LearningZhihui Zhang, Shiliang Sun, Jing Zhao, Tengfei Song et al.EMNLP Findings 2025
- InImageTrans: Multimodal LLM-based Text Image Machine TranslationFei Zuo, Kehai Chen, Yu Zhang, Zhengshan Xue et al.ACL Findings 2025 · 1 citation
