Lune

ICLR2025Top-tier venue

Better Instruction-Following Through Minimum Bayes Risk

Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Khoshfetrat Pakazad, Graham Neubig

Abstract

General-purpose LLM judges capable of human-level evaluation provide not only a scalable and accurate way of evaluating instruction-following LLMs but also new avenues for supervising and improving their performance. One promising way of leveraging LLM judges for supervision is through Minimum Bayes Risk (MBR) decoding, which uses a reference-based evaluator to select a high-quality output from amongst a set of candidate outputs. In the first part of this work, we explore using MBR decoding as a method for improving the test-time performance of instruction-following LLMs. We find that MBR decoding with reference-based LLM judges substantially improves over greedy decoding, best-of-N decoding with reference-free judges and MBR decoding with lexical and embedding-based metrics on AlpacaEval and MT-Bench. These gains are consistent across LLMs with up to 70B parameters, demonstrating that smaller LLM judges can be used to supervise much larger LLMs. Then, seeking to retain the improvements from MBR decoding while mitigating additional test-time costs, we explore iterative self-training on MBR-decoded outputs. We find that self-training using Direct Preference Optimisation leads to significant performance gains, such that the selftrained models with greedy decoding generally match and sometimes exceed the performance of their base models with MBR decoding.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

3 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Do NOT Think That Much for 2+3=? On the Overthinking of Long Reasoning Models cites Tree of Thoughts: Deliberate Problem Solving with Large Language Models
  • Do NOT Think That Much for 2+3=? On the Overthinking of Long Reasoning Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Tree of Thoughts: Deliberate Problem Solving with Large Language Models cites Self-Refine: Iterative Refinement with Self-Feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers3

  1. Thoughts Are All Over the Place: On the Underthinking of Long Reasoning ModelsYue Wang, Qiuzhi Liu, Jiahao Xu, Tian Liang et al.NeurIPS 2025 · 13 citations
  2. Task-Awareness Improves LLM Generations and UncertaintyTim Tomov, Dominik Fuchsgruber, Stephan GünnemannICML 2026 · 2 citations
  3. Do NOT Think That Much for 2+3=? On the Overthinking of Long Reasoning ModelsXingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He et al.ICML 2025

Builds on31

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. Tree of Thoughts: Deliberate Problem Solving with Large Language ModelsShunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran et al.NeurIPS 2023 · 5,068 citations
  5. Self-Refine: Iterative Refinement with Self-FeedbackAman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan et al.NeurIPS 2023 · 4,972 citations

Related papers

  1. Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li et al.ICML 2024 · 569 citations
  2. MBR and QE Finetuning: Training-time Distillation of the Best and Most Expensive Decoding MethodsMara Finkelstein, Markus FreitagICLR 2024 · 39 citations
  3. Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-JudgeTianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu et al.EMNLP 2025 · 6 citations
  4. Thinking LLMs: General Instruction Following with Thought GenerationTianhao Wu, Janice Lan, Weizhe Yuan, Jiantao Jiao et al.ICML 2025 · 3 citations
  5. An Empirical Study of Translation Hypothesis Ensembling with Large Language ModelsAntónio Farinhas, José Guilherme Camargo de Souza, André F. T. MartinsEMNLP 2023 · 4 citations
Dusk over the sea between two cliffs drawn in fine vertical lines