Lune

ACL Findings2024

Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation

Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, Min Zhang, Jinsong Su

Abstract

In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this regard, conventional cascaded methods suffer from issues such as error propagation, massive parameters, and difficulties in deployment and retaining visual characteristics of the input image. Thus, constructing end-to-end models has become an option, which, however, faces two main challenges: 1) the huge modeling burden, as it is required to simultaneously learn alignment across languages and preserve the visual characteristics of the input image; 2) the difficulties of directly predicting excessively lengthy pixel sequences. In this paper, we propose Translatotron-V(ision), an end-to-end IIMT model consisting of four modules. In addition to an image encoder, and an image decoder, our model contains a target text decoder and an image tokenizer. Among them, the target text decoder is used to alleviate the language alignment burden, and the image tokenizer converts long sequences of pixels into shorter sequences of visual tokens, preventing the model from focusing on low-level visual features. Besides, we present a two-stage training framework for our model to assist the model in learning alignment across modalities and languages. Finally, we propose a locationaware evaluation metric called Structure-BLEU to assess the translation quality of the generated images. Experimental results demonstrate that our model achieves competitive performance compared to cascaded models with only 70.9% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation cites Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
  • PRIM: Towards Practical In-Image Multilingual Machine Translation cites Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
  • PRIM: Towards Practical In-Image Multilingual Machine Translation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Rethinking and Improving Relative Position Encoding for Vision Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine TranslationYupu Liang, Yaping Zhang, Zhiyang Zhang, Yang Zhao et al.ACL 2025 · 6 citations
  2. MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine TranslationGengluo Li, Chengquan Zhang, Yupu Liang, Huawen Shen et al.CVPR 2026 · 6 citations
  3. PRIM: Towards Practical In-Image Multilingual Machine TranslationYanzhi Tian, Zeming Liu, Zhengyang Liu, Chong Feng et al.EMNLP 2025 · 4 citations
  4. Empowering Backbone Models for Visual Text Generation with Input Granularity Control and Glyph-Aware TrainingWenbo Li, Guohao Li, Zhibin Lan, Xue Xu et al.EMNLP 2024 · 2 citations
  5. VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image TranslationBo Li, Ronghao Chen, Ningyuan Deng, Huacan Wang et al.KDD 2026

Builds on10

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Zero-Shot Text-to-Image GenerationAditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray et al.ICML 2021 · 6,356 citations
  3. TrOCR: Transformer-Based Optical Character Recognition with Pre-trained ModelsMinghao Li, Tengchao Lv, Jingye Chen, Lei Cui et al.AAAI 2023 · 607 citations
  4. Rethinking and Improving Relative Position Encoding for Vision TransformerKan Wu, Houwen Peng, Minghao Chen, Jianlong Fu et al.ICCV 2021 · 427 citations
  5. A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine TranslationYongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou et al.ACL 2020 · 145 citations

Related papers

  1. In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence ModelYanzhi Tian, Xiang Li, Zeming Liu, Yuhang Guo et al.EMNLP Findings 2023 · 3 citations
  2. PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image TranslationShaolin Zhu, Shangjie Li, Yikun Lei, Deyi XiongACL 2023 · 12 citations
  3. Probing Multi-modal Machine Translation with Pre-trained Language ModelYawei Kong, Kai FanACL Findings 2021 · 13 citations
  4. CCIM: Cross-modal Cross-lingual Interactive Image TranslationCong Ma, Yaping Zhang, Mei Tu, Yang Zhao et al.EMNLP Findings 2023 · 3 citations
  5. Translatotron 2: High-quality direct speech-to-speech translation with voice preservationYe Jia, Michelle Tadmor Ramanovich, Tal Remez, Roi PomerantzICML 2022 · 107 citations
Dusk over the sea between two cliffs drawn in fine vertical lines