Lune

EMNLP Findings2025

RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation

Tianjiao Li, Mengran Yu, Chenyu Shi, Yanjun Zhao, Xiaojing Liu, Qi Zhang, Xuanjing Huang, Qiang Zhang, Jiayin Wang

Abstract

Large language models (LLMs) possess strong multilingual capabilities, and combining Reinforcement Learning from Human Feedback (RLHF) with translation tasks has shown great potential. However, we observe that this paradigm performs unexpectedly poorly when applied to colloquial subtitle translation tasks. In this work, we investigate this issue and find that the offline reward model (RM) gradually diverges from the online LLM due to distributional shift, ultimately leading to undesirable training outcomes. To address this, we propose RIVAL, an adversarial training framework that formulates the process as a min-max game between the RM and the LLM. RIVAL iteratively updates the both models, with the RM trained to distinguish strong from weak translations (qualitative preference reward), and the LLM trained to enhance its translation for closing this gap. To stabilize training and improve generalizability, we also incorporate quantitative preference reward (e.g., BLEU) into the RM, enabling reference-free quality modeling aligned with human evaluation. Through extensive experiments, we demonstrate that the proposed training framework significantly improves upon translation baselines. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation cites Training language models to follow instructions with human feedback
  • M²PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation cites Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
  • M²PO: Multi-Perspective Multi-Pair Preference Optimization for Machine Translation cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. M²PO: Multi-Perspective Multi-Pair Preference Optimization for Machine TranslationHao Wang, Linlong Xu, Heng Liu, Yangyang Liu et al.ACL 2026

Builds on19

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timeMitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs et al.ICML 2022 · 1,464 citations
  3. Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine TranslationHaoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan et al.ICML 2024 · 447 citations
  4. Adversarial Policies: Attacking Deep Reinforcement LearningAdam Gleave, Michael Dennis, Cody Wild, Neel Kant et al.ICLR 2020 · 415 citations
  5. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned ModelsAlexander Pan, Kush Bhatia, Jacob SteinhardtICLR 2022 · 293 citations

Related papers

  1. From Utterance to Vividity: Training Expressive Subtitle Translation LLM via Adaptive Local Preference OptimizationChaoqun Cui, Shijing Wang, Liangbin Huang, Qingqing Gu et al.ICLR 2026 · 1 citation
  2. Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM GamePengyu Cheng, Yifan Yang, Jian Li, Yong Dai et al.ACL Findings 2024 · 6 citations
  3. Mending the Holes: Mitigating Reward Hacking in Reinforcement Learning for Multilingual TranslationYifeng Liu, Siqi Ouyang, Yatish Hosmane Revanasiddappa, Lei LiACL Findings 2026
  4. Real-Time Aligned Reward Model beyond SemanticsZixuan Huang, Xin Xia, Yuxi Ren, Jianbin Zheng et al.ICML 2026 · 18 citations
  5. A Regret Minimization Framework on Preference Learning in Large Language ModelsSuhwan Kim, Taehyun Cho, Youngsoo Jang, Geon-Hyeong Kim et al.ICML 2026
Dusk over the sea between two cliffs drawn in fine vertical lines