Lune

ACL Findings2025

InImageTrans: Multimodal LLM-based Text Image Machine Translation

Fei Zuo, Kehai Chen, Yu Zhang, Zhengshan Xue, Min Zhang

Abstract

Multimodal large language models (MLLMs) have shown remarkable capabilities across various downstream tasks. However, when MLLMs are transferred to the text image machine translation (TiMT) task, preliminary experiments reveal that MLLMs suffer from serious repetition and omission hallucinations. To alleviate these issues, this paper first designs an efficient MLLM named InImageTrans for TiMT and then proposes a simple and effective method named multi-conditional direct preference optimization (mcDPO) for advancing the TiMT. Particularly, the proposed mcDPO not only guides the MLLM in rejecting repetition output by creating text output preference pairs automatically, but also guides the MLLM in paying more attention to text information in images by creating image input preference pairs. Furthermore, we build a high-quality benchmark called MCiT for comprehensively evaluating the TiMT capabilities of InImageTrans. Experimental results show that the proposed method significantly outperforms existing open-source MLLMs on MCiT. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

4 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Preference Ranking Optimization for Human Alignment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation cites Learning Transferable Visual Models From Natural Language Supervision
  • CogVLM: Visual Expert for Pretrained Language Models cites PaLM-E: An Embodied Multimodal Language Model
  • First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-Training cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers4

  1. First SFT, Second RL, Third UPT: Continual Improving Multi-Modal LLM Reasoning via Unsupervised Post-TrainingLai Wei, Yuting Li, Chen Wang, Yue Wang et al.NeurIPS 2025 · 28 citations
  2. LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video GenerationXiangqing Zheng, CHENGYUE WU, Kehai Chen, Min zhangICML 2026 · 3 citations
  3. Paraphrasing as Zero-shot Translation with Feature-guided Diversity EnhancementZiyue Yan, Hongying Zan, Xinglin Lyu, Hongfei XuACL 2026
  4. MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE InferenceBo Li, Chuan Wu, Shaolin ZhuACL 2026

Builds on23

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. PaLM-E: An Embodied Multimodal Language ModelDanny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch et al.ICML 2023 · 2,601 citations
  4. CogVLM: Visual Expert for Pretrained Language ModelsWeihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong et al.NeurIPS 2024 · 858 citations
  5. Preference Ranking Optimization for Human AlignmentFeifan Song, Bowen Yu, Minghao Li, Haiyang Yu et al.AAAI 2024 · 357 citations

Related papers

  1. Make Imagination Clearer! Stable Diffusion-based Visual Imagination for Multimodal Machine TranslationAndong Chen, Yuchen Song, Kehai Chen, Xuefeng Bai et al.ACL 2025
  2. CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMsJinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen et al.ICLR 2025
  3. Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference OptimizationYuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang et al.ACL Findings 2025 · 6 citations
  4. Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMsXudong Li, Mengdan Zhang, Peixian Chen, Xiawu Zheng et al.NeurIPS 2025 · 4 citations
  5. mDPO: Conditional Preference Optimization for Multimodal Large Language ModelsFei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu et al.EMNLP 2024 · 11 citations
Dusk over the sea between two cliffs drawn in fine vertical lines