Lune

EMNLP Findings2025

DORM: Preference Data Weights Optimization for Reward Modeling in LLM Alignment

Rongzhi Zhang, Chenwei Zhang, Xinyang Zhang, Liang Qiu, Haoming Jiang, Yuchen Zhuang, Qingru Zhang, Hyokun Yun, Xian Li, Bing Yin, Tuo Zhao, Chao Zhang

Abstract

Aligning large language models (LLMs) with human preferences relies heavily on high-quality reward models. However, existing approaches struggle with two critical challenges: noisy preference labels and the varying importance of preference samples. We introduce DORM, a method that enhances reward modeling by learning to dynamically weigh preference data. DORM initializes data importance using a combination of model uncertainty and prediction disagreement, then iteratively refines them via bilevel optimization to maximize validation performance. Using only 50k samples, DORM trains a 12B reward model that achieves 90.5% accuracy on RewardBench, matching the performance of models trained on significantly larger datasets. Furthermore, downstream alignment tasks show that fine-tuned LLMs with DORM achieve a 61.2% win rate against baseline methods, highlighting its data efficiency and generalizability. Recent studies highlight two key challenges in reward modeling for LLM alignment: (1) Preference data are often noisy: Zheng et al. (2023) report 19-37% preferences provided by crowd workers are noisy. Gao et al. (2024) reveal that preference noise is observed in a wide range of tasks, including QA, Summarization, and Dialogue, where the noise rate ranges between 20-40%. (2) Not all preference data contributes equally: Noisy preference data forces reward models to require more training data to achieve desired performance. For instance, Wang et al. (2024a) categorize preference data into groups based on preference strength and find that some groups negatively impact reward modeling, which highlights the need for selective data curation to boost reward model performance.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • ActiveUltraFeedback: Efficient Preference Data Generation using Active Learning cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. ActiveUltraFeedback: Efficient Preference Data Generation using Active LearningDavit Melikidze, Marian Schneider, Jessica Lam, Martin Wertich et al.ICML 2026

Builds on15

  1. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  2. Defining and Characterizing Reward GamingJoar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, David KruegerNeurIPS 2022 · 466 citations
  3. Bilevel Optimization: Convergence Analysis and Enhanced DesignKaiyi Ji, Junjie Yang, Yingbin LiangICML 2021 · 343 citations
  4. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise ComparisonsBanghua Zhu, Michael I. Jordan, Jiantao JiaoICML 2023 · 273 citations
  5. Learning Fast Sample Re-weighting Without Reward DataZizhao Zhang, Tomas PfisterICCV 2021 · 109 citations

Related papers

  1. RMO: Towards Better LLM Alignment via Reshaping Reward Margin DistributionsYanchi Ru, Yue Huang, Xiangliang ZhangAAAI 2026 · 1 citation
  2. What Matters in Data for DPO?Yu Pan, Zhongze Cai, Huaiyang Zhong, Guanting Chen et al.NeurIPS 2025 · 13 citations
  3. Less is More: Improving LLM Alignment via Preference Data SelectionXun Deng, Han Zhong, Rui Ai, Fuli Feng et al.NeurIPS 2025 · 4 citations
  4. Bootstrapping Language Models with DPO Implicit RewardsChangyu Chen, Zichen Liu, Chao Du, Tianyu Pang et al.ICLR 2025
  5. ActiveDPO: Active Direct Preference Optimization for Sample-Efficient AlignmentXiaoqiang Lin, Arun Verma, Zhongxiang Dai, Daniela Rus et al.ICLR 2026 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines