ACL Findings2024
DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling
Abstract
The performance of the reward model (RM) is a critical factor in improving the effectiveness of the large language model (LLM) during alignment fine-tuning. There remain two challenges in RM training: 1) training the same RM using various categories of data may cause its generalization performance to suffer from multi-task disturbance, and 2) the human annotation consistency rate is generally only 60% to 75%, causing training data to contain a lot of noise. To tackle these two challenges, we introduced the idea of Mixture-of-Experts (MoE) into the field of RM for the first time. We propose the Double-Layer MoE RM (DMoERM). The outer layer MoE is a sparse model. After classifying an input into task categories, we route it to the corresponding inner layer task-specific model. The inner layer MoE is a dense model. We decompose the specific task into multiple capability dimensions and individually fine-tune a LoRA expert on each one. Their outputs are then synthesized by an MLP to compute the final rewards. To minimize costs, we call a public LLM API to obtain the capability preference labels. The validation on manually labeled datasets confirms that our model attains superior consistency with human preference and outstrips advanced generative approaches. Meanwhile, through BoN sampling and RL experiments, we demonstrate that our model outperforms state-of-the-art ensemble methods of RM and mitigates the overoptimization problem. Our code is available at: https://github.com/quanshr/DMoERM.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- LoRA: Low-Rank Adaptation of Large Language Models cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- GLaM: Efficient Scaling of Language Models with Mixture-of-Experts cites GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Automatically Generating Numerous Context-Driven SFT Data for LLMs Across Diverse Granularity cites Training language models to follow instructions with human feedback
- LASeR: Learning to Adaptively Select Reward Models with Multi-Arm Bandits cites LoRA: Low-Rank Adaptation of Large Language Models
- LASeR: Learning to Adaptively Select Reward Models with Multi-Arm Bandits cites Training language models to follow instructions with human feedback
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models cites Training language models to follow instructions with human feedback
Cited by top-tier papers3
- Automatically Generating Numerous Context-Driven SFT Data for LLMs Across Diverse GranularityShanghaoran QuanAAAI 2025 · 7 citations
- MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference LearningJingyan Shen, Jiarui Yao, Rui Yang, Yifan Sun et al.EMNLP 2025 · 2 citations
- LASeR: Learning to Adaptively Select Reward Models with Multi-Arm BanditsDuy Nguyen, Archiki Prasad, Elias Stengel-Eskin, Mohit BansalNeurIPS 2025
Builds on25
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
- LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
- GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingDmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen et al.ICLR 2021 · 1,954 citations
- GLaM: Efficient Scaling of Language Models with Mixture-of-ExpertsNan Du, Yanping Huang, Andrew M. Dai, Simon Tong et al.ICML 2022 · 1,173 citations
Related papers
- Routing Manifold Alignment Improves Generalization of Mixture-of-Experts LLMsZhongyang Li, Ziyue Li, Tianyi ZhouICLR 2026 · 5 citations
- From Outcome to Process: Optimizing MoE Load Balancing with MCTSWenjun Ke, Hengyuan Xu, Ziyu Shang, Yao He et al.ACL Findings 2026
- CoPL: Collaborative Preference Learning for Personalizing LLMsYoungbin Choi, Seunghyuk Cho, Minjong Lee, MoonJeong Park et al.EMNLP 2025
- Multi-objective Large Language Model Alignment with Hierarchical ExpertsZhuo Li, Guodong DU, Weiyang Guo, Yigeng Zhou et al.ICLR 2026 · 17 citations
- The Devil Is in the Details: Tackling Unimodal Spurious Correlations for Generalizable Multimodal Reward ModelsZichao Li, Xueru Wen, Jie Lou, Yuqiu Ji et al.ICML 2025
