Lune

ACL Findings2024

Exploring Domain Robust Lightweight Reward Models based on Router Mechanism

Hyuk Namgoong, Jeesu Jung, Sangkeun Jung, Yoon-Hyung Roh

Abstract

Recent advancements in large language models have heavily relied on the large reward model from reinforcement learning from human feedback for fine-tuning. However, the use of a single reward model across various domains may not always be optimal, often requiring retraining from scratch when new domain data is introduced. To address these challenges, we explore the utilization of small language models operating in a domain-specific manner based on router mechanisms. Our three approaches are: 1) utilize mixture of experts to form a single reward model by modularizing an internal router and experts, 2) employing external router to select the appropriate reward model from multiple domain-specific models, and 3) the framework reduces parameter size by loading reward models and router adapters onto a single small language model using adapters. Experimental validation underscores the effectiveness of our approach, demonstrating performance comparable to baseline methods while also reducing the total parameter size.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • QLoRA: Efficient Finetuning of Quantized LLMs cites Training language models to follow instructions with human feedback
  • Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Sampling cites LoRA: Low-Rank Adaptation of Large Language Models
  • Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Sampling cites Training language models to follow instructions with human feedback
  • Training Diffusion Models with Reinforcement Learning cites Training language models to follow instructions with human feedback
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N SamplingJizhou Guo, Zhaomin Wu, Hanchen Yang, Philip S. YuKDD 2026

Builds on10

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  4. Training Diffusion Models with Reinforcement LearningKevin Black, Michael Janner, Yilun Du, Ilya Kostrikov et al.ICLR 2024 · 816 citations
  5. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding SharingPengcheng He, Jianfeng Gao, Weizhu ChenICLR 2023 · 394 citations

Related papers

  1. Nexus: Adaptive Upcycling to Efficiently Pretrain Mixture of ExpertsNikolas Gritsch, Qizhen Zhang, Acyr Locatelli, Sara Hooker et al.EMNLP Findings 2025 · 1 citation
  2. LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention RoutingWenbing Li, Zikai Song, Hang Zhou, Junqing Yu et al.ICLR 2026 · 20 citations
  3. MoLoRA: Composable Specialization via Per-Token Adapter RoutingShrey Shah, Justin WagleICML 2026 · 3 citations
  4. Prototypical Reward Network for Data-Efficient RLHFJinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen et al.ACL 2024
  5. LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA ExpertsYuan Zhuang, Yi Shen, Yuexin Bian, Qing Su et al.ICLR 2026 · 15 citations
Dusk over the sea between two cliffs drawn in fine vertical lines