EMNLP Findings2024
AnyTrans: Translate AnyText in the Image with Large Scale Models
Zhipeng Qian, Pei Zhang, Baosong Yang, Kai Fan, Yiwei Ma, Derek F. Wong, Xiaoshuai Sun, Rongrong Ji
Abstract
This paper introduces AnyTrans, an allencompassing framework for the task-In-Image Machine Translation (IIMT), which includes multilingual text translation and text fusion within images. Our framework leverages the strengths of large-scale models, such as Large Language Models (LLMs) and textguided diffusion models, to incorporate contextual cues from both textual and visual elements during translation. The few-shot learning capability of LLMs allows for the translation of fragmented texts by considering the overall context. Meanwhile, diffusion models' advanced inpainting and editing abilities make it possible to fuse translated text seamlessly into the original image while preserving its style and realism. Our framework can be constructed entirely using open-source models and requires no training, making it highly accessible and easily expandable. To encourage advancement in the IIMT task, we have meticulously compiled a test dataset called MTIT6, which consists of multilingual text image translation data from six language pairs.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
4 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine Translation cites Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
- Adding Conditional Control to Text-to-Image Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
- Adding Conditional Control to Text-to-Image Diffusion Models cites GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- Adding Conditional Control to Text-to-Image Diffusion Models cites T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models cites Adding Conditional Control to Text-to-Image Diffusion Models
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models cites High-Resolution Image Synthesis with Latent Diffusion Models
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models cites GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- PRIM: Towards Practical In-Image Multilingual Machine Translation cites Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
- PRIM: Towards Practical In-Image Multilingual Machine Translation cites High-Resolution Image Synthesis with Latent Diffusion Models
Cited by top-tier papers4
- Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine TranslationYupu Liang, Yaping Zhang, Zhiyang Zhang, Yang Zhao et al.ACL 2025 · 6 citations
- MMTIT-Bench: A Multilingual and Multi-Scenario Benchmark with Cognition-Perception-Reasoning Guided Text-Image Machine TranslationGengluo Li, Chengquan Zhang, Yupu Liang, Huawen Shen et al.CVPR 2026 · 6 citations
- PRIM: Towards Practical In-Image Multilingual Machine TranslationYanzhi Tian, Zeming Liu, Zhengyang Liu, Chong Feng et al.EMNLP 2025 · 4 citations
- VaaWIT: Visual-Aware Adaptation of Large Language Models for Multilingual Web Image TranslationBo Li, Ronghao Chen, Ningyuan Deng, Huacan Wang et al.KDD 2026
Builds on21
- High-Resolution Image Synthesis with Latent Diffusion ModelsRobin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser et al.CVPR 2022 · 13,123 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Adding Conditional Control to Text-to-Image Diffusion ModelsLvmin Zhang, Anyi Rao, Maneesh AgrawalaICCV 2023 · 6,759 citations
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion ModelsAlexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam et al.ICML 2022 · 4,691 citations
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion ModelsChong Mou, Xintao Wang, Liangbin Xie, Yanze Wu et al.AAAI 2024 · 1,641 citations
Related papers
- AnyText: Multilingual Visual Text Generation and EditingYuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng et al.ICLR 2024 · 148 citations
- Exploring In-Image Machine Translation with Real-World BackgroundYanzhi Tian, Zeming Liu, Zhengyang Liu, Yuhang GuoACL Findings 2025
- De-Diffusion Makes Text a Strong Cross-Modal InterfaceChen Wei, Chenxi Liu, Siyuan Qiao, Zhishuai Zhang et al.CVPR 2024
- MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image GenerationMarco Bellagente, Manuel Brack, Hannah Teufel, Felix Friedrich et al.NeurIPS 2023 · 31 citations
- InImageTrans: Multimodal LLM-based Text Image Machine TranslationFei Zuo, Kehai Chen, Yu Zhang, Zhengshan Xue et al.ACL Findings 2025 · 1 citation
