Lune

EMNLP Findings2024

LAMBDA: Large Language Model-Based Data Augmentation for Multi-Modal Machine Translation

Yusong Wang, Dongyuan Li, Jialun Shen, Yicheng Xu, Mingkun Xu, Kotaro Funakoshi, Manabu Okumura

Abstract

Multi-modal machine translation (MMT) can reduce ambiguity and semantic distortion compared with traditional machine translation (MT) by utilizing auxiliary information such as images. However, current MMT methods face two primary challenges. The first is their underperformance compared to MT methods based on pre-trained models. The second is the inadequate exploitation and integration of the image modality within the model, primarily due to a lack of triplet training data. A mainstream approach is to introduce large amounts of parallel and monolingual data to train the text model and the visual model separately. However, incorporating extensive external data can result in data imbalance, which may introduce biases during training. Additionally, the collection and cleaning of such large datasets is labor-intensive. To overcome these challenges, we introduce a novel, low-cost, large language model-based data augmentation method called LAMBDA, which can enrich the original samples and expand the dataset without requiring external images and text. We propose a finegrained image captioning module with a noise filter to hierarchically and accurately extract unexploited information from images. Additionally, we design two specific prompts to guide the GPT-3.5 model in generating enriched texts and the corresponding translations. The enriched samples contain diverse text and strong connections between text and images, leading to significant improvements for MMT baselines, with the highest being an increase of up to 3.83 BLEU score and 3.61 METEOR score.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction cites Generating Training Data with Language Models: Towards Zero-Shot Language Understanding
  • SHIFT: Selected Helpful Informative Frame for Video-guided Machine Translation cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translation cites Learning Transferable Visual Models From Natural Language Supervision
  • DART: Disambiguation-Aware Reasoning for Video-guided Machine Translation cites SHIFT: Selected Helpful Informative Frame for Video-guided Machine Translation
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. SHIFT: Selected Helpful Informative Frame for Video-guided Machine TranslationBoyu Guan, Chuang Han, Yining Zhang, Yupu Liang et al.EMNLP 2025
  2. Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from TranslationJulian Spravil, Sebastian Houben, Sven BehnkeAAAI 2026
  3. DART: Disambiguation-Aware Reasoning for Video-guided Machine TranslationBoyu Guan, Chuang Han, Yang Zhao, Chengqing ZongACL 2026

Builds on14

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Denoising Diffusion Probabilistic ModelsJonathan Ho, Ajay Jain, Pieter AbbeelNeurIPS 2020 · 35,902 citations
  3. Generating Training Data with Language Models: Towards Zero-Shot Language UnderstandingYu Meng, Jiaxin Huang, Yu Zhang, Jiawei HanNeurIPS 2022 · 309 citations
  4. Repair Is Nearly Generation: Multilingual Program Repair with LLMsHarshit Joshi, José Pablo Cambronero Sánchez, Sumit Gulwani, Vu Le et al.AAAI 2023 · 182 citations
  5. RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet ExtractionYew Ken Chia, Lidong Bing, Soujanya Poria, Luo SiACL Findings 2022 · 103 citations

Related papers

  1. VQA-Augmented Machine Translation with Cross-Modal Contrastive LearningZhihui Zhang, Shiliang Sun, Jing Zhao, Tengfei Song et al.EMNLP Findings 2025
  2. Probing Multi-modal Machine Translation with Pre-trained Language ModelYawei Kong, Kai FanACL Findings 2021 · 13 citations
  3. Multimodal Machine Translation with Text-Image In-depth QuestioningYue Gao, Jing Zhao, Shiliang Sun, Xiaosong Qiao et al.ACL Findings 2025
  4. Beyond Triplet: Leveraging the Most Data for Multimodal Machine TranslationYaoming Zhu, Zewei Sun, Shanbo Cheng, Luyang Huang et al.ACL Findings 2023 · 14 citations
  5. Incorporating Probing Signals into Multimodal Machine Translation via Visual Question-Answering PairsYuxin Zuo, Bei Li, Chuanhao Lv, Tong Zheng et al.EMNLP Findings 2023
Dusk over the sea between two cliffs drawn in fine vertical lines