Lune

ACL Findings2025

Exploring In-Image Machine Translation with Real-World Background

Yanzhi Tian, Zeming Liu, Zhengyang Liu, Yuhang Guo

Abstract

In-Image Machine Translation (IIMT) aims to translate texts within images from one language to another. Previous research on IIMT was primarily conducted on simplified scenarios such as images of one-line text with black font in white backgrounds, which is far from reality and impractical for applications in the real world. To make IIMT research practically valuable, it is essential to consider a complex scenario where the text backgrounds are derived from real-world images. To facilitate research of complex scenarios IIMT, we design an IIMT dataset that includes subtitle text with a real-world background. However, previous IIMT models perform inadequately in complex scenarios. To address the issue, we propose the DebackX model, which separates the background and text-image from the source image, performs translation on the text-image directly, and fuses the translated text-image with the background to generate the target image. Experimental results show that our model achieves improvements in both translation quality and visual effect. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

2 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Vector-quantized Image Modeling with Improved VQGAN cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • MaskGIT: Masked Generative Image Transformer cites Vector-quantized Image Modeling with Improved VQGAN
  • MaskGIT: Masked Generative Image Transformer cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • PRIM: Towards Practical In-Image Multilingual Machine Translation cites Vector-quantized Image Modeling with Improved VQGAN
  • PRIM: Towards Practical In-Image Multilingual Machine Translation cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • PRIM: Towards Practical In-Image Multilingual Machine Translation cites AnyText: Multilingual Visual Text Generation and Editing
  • Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction cites Vector-quantized Image Modeling with Improved VQGAN
  • Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction cites An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
  • Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation Evaluation cites PRIM: Towards Practical In-Image Multilingual Machine Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers2

  1. PRIM: Towards Practical In-Image Multilingual Machine TranslationYanzhi Tian, Zeming Liu, Zhengyang Liu, Chong Feng et al.EMNLP 2025 · 4 citations
  2. Beyond Literal Mapping: Benchmarking and Improving Non-Literal Translation EvaluationYanzhi Tian, Cunxiang Wang, Zeming Liu, Heyan Huang et al.ACL 2026 · 3 citations

Builds on15

  1. An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAlexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn et al.ICLR 2021 · 21,477 citations
  2. Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale PredictionKeyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng et al.NeurIPS 2024 · 1,199 citations
  3. Vector-quantized Image Modeling with Improved VQGANJiahui Yu, Xin Li, Jing Yu Koh, Han Zhang et al.ICLR 2022 · 753 citations
  4. MaskGIT: Masked Generative Image TransformerHuiwen Chang, Han Zhang, Lu Jiang, Ce Liu et al.CVPR 2022 · 346 citations
  5. AnyText: Multilingual Visual Text Generation and EditingYuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng et al.ICLR 2024 · 148 citations

Related papers

  1. In-Image Neural Machine Translation with Segmented Pixel Sequence-to-Sequence ModelYanzhi Tian, Xiang Li, Zeming Liu, Yuhang Guo et al.EMNLP Findings 2023 · 3 citations
  2. PATIMT-Bench: A Multi-Scenario Benchmark for Position-Aware Text Image Machine Translation in Large Vision-Language ModelsWanru Zhuang, Wenbo Li, Zhibin Lan, Xu Han et al.EMNLP Findings 2025
  3. AnyTrans: Translate AnyText in the Image with Large Scale ModelsZhipeng Qian, Pei Zhang, Baosong Yang, Kai Fan et al.EMNLP Findings 2024 · 3 citations
  4. Exploring Better Text Image Translation with Multimodal CodebookZhibin Lan, Jiawei Yu, Xiang Li, Wen Zhang et al.ACL 2023 · 12 citations
  5. CCIM: Cross-modal Cross-lingual Interactive Image TranslationCong Ma, Yaping Zhang, Mei Tu, Yang Zhao et al.EMNLP Findings 2023 · 3 citations
Dusk over the sea between two cliffs drawn in fine vertical lines