Lune

EMNLP Findings2023

Epsilon Sampling Rocks: Investigating Sampling Strategies for Minimum Bayes Risk Decoding for Machine Translation

Markus Freitag, Behrooz Ghorbani, Patrick Fernandes

Abstract

Recent advances in machine translation (MT) have shown that Minimum Bayes Risk (MBR) decoding can be a powerful alternative to beam search decoding, especially when combined with neural-based utility functions. However, the performance of MBR decoding depends heavily on how and how many candidates are sampled from the model. In this paper, we explore how different sampling approaches for generating candidate lists for MBR decoding affect performance. We evaluate popular sampling approaches, such as ancestral, nucleus, and top-k sampling. Based on our insights into their limitations, we experiment with the recently proposed epsilon-sampling (Hewitt et al., 2022) approach, which prunes away all tokens with a probability smaller than epsilon, ensuring that each token in a sample receives a fair probability mass. Through extensive human evaluations, we demonstrate that MBR decoding based on epsilon-sampling significantly outperforms not only beam search decoding, but also MBR decoding with all other tested sampling methods across four language pairs.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

26 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms cites Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation
  • Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms cites MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods
  • Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms cites COMET: A Neural Framework for MT Evaluation
  • Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms cites BLEU might be Guilty but References are not Innocent
  • Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms cites The Curious Case of Neural Text Degeneration
  • Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms cites Truncation Sampling as Language Model Desmoothing
  • Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine Translation cites COMET: A Neural Framework for MT Evaluation
  • MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding Methods cites Truncation Sampling as Language Model Desmoothing
  • Closing the Curious Case of Neural Text Degeneration cites The Curious Case of Neural Text Degeneration
  • Closing the Curious Case of Neural Text Degeneration cites Truncation Sampling as Language Model Desmoothing
  • Truncation Sampling as Language Model Desmoothing cites The Curious Case of Neural Text Degeneration
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers26

  1. Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel CollaborationYichong Huang, Xiaocheng Feng, Baohang Li, Yang Xiang et al.NeurIPS 2024 · 94 citations
  2. MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding MethodsMara Finkelstein, Markus FreitagICLR 2024 · 39 citations
  3. Closing the Curious Case of Neural Text DegenerationMatthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta et al.ICLR 2024 · 31 citations
  4. QUEST: Quality-Aware Metropolis-Hastings Sampling for Machine TranslationGonçalo Rui Alves Faria, Sweta Agrawal, António Farinhas, Ricardo Rei et al.NeurIPS 2024 · 23 citations
  5. Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion AlgorithmsFiras Trabelsi, David Vilar, Mara Finkelstein, Markus FreitagNeurIPS 2024 · 18 citations

Builds on5

  1. The Curious Case of Neural Text DegenerationAri Holtzman, Jan Buys, Li Du, Maxwell Forbes et al.ICLR 2020 · 4,112 citations
  2. Truncation Sampling as Language Model DesmoothingJohn Hewitt, Christopher D. Manning, Percy LiangEMNLP Findings 2022 · 26 citations
  3. BLEU might be Guilty but References are not InnocentMarkus Freitag, David Grangier, Isaac CaswellEMNLP 2020 · 13 citations
  4. COMET: A Neural Framework for MT EvaluationRicardo Rei, Craig Stewart, Ana C. Farinha, Alon LavieEMNLP 2020 · 6 citations
  5. Understanding the Properties of Minimum Bayes Risk Decoding in Neural Machine TranslationMathias Müller, Rico SennrichACL 2021

Related papers

  1. Model-Based Minimum Bayes Risk Decoding for Text GenerationYuu Jinnai, Tetsuro Morimura, Ukyo Honda, Kaito Ariu et al.ICML 2024 · 9 citations
  2. Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine TranslationBoxuan Lyu, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu OkumuraACL 2025
  3. Quality-Aware Translation Models: Efficient Generation and Quality Estimation in a Single ModelChristian Tomani, David Vilar, Markus Freitag, Colin Cherry et al.ACL 2024
  4. Follow the Wisdom of the Crowd: Effective Text Generation via Minimum Bayes Risk DecodingMirac Suzgun, Luke Melas-Kyriazi, Dan JurafskyACL Findings 2023 · 12 citations
  5. Hyperparameter-Free Approach for Faster Minimum Bayes Risk DecodingYuu Jinnai, Kaito AriuACL Findings 2024
Dusk over the sea between two cliffs drawn in fine vertical lines