Lune

ACL Findings2024

TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedback

Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Wontae Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, Chang Dong Yoo

Abstract

Reinforcement Learning from Human Feedback (RLHF) leverages human preference data to train language models to align more closely with human essence. These human preference data, however, are labeled at the sequence level, creating a mismatch between sequence-level preference labels and tokens, which are autoregressively generated from the language model. Although several recent approaches have tried to provide token-level (i.e., dense) rewards for each individual token, these typically rely on predefined discrete reward values (e.g., positive: +1, negative: -1, neutral: 0), failing to account for varying degrees of preference inherent to each token. To address this limitation, we introduce TLCR (Token-Level Continuous Reward) for RLHF, which incorporates a discriminator trained to distinguish positive and negative tokens, and the confidence of the discriminator is used to assign continuous rewards to each token considering the context. Extensive experiments show that TLCR leads to consistent performance improvements over previous sequence-level or token-level discrete rewards on open-ended generation benchmarks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

19 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Scaling Laws for Reward Model Overoptimization cites Training language models to follow instructions with human feedback
  • Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual Grounding cites Training language models to follow instructions with human feedback
  • Fine-Grained Human Feedback Gives Better Rewards for Language Model Training cites Training language models to follow instructions with human feedback
  • T-REG: Preference Optimization with Token-Level Reward Regularization cites Scaling Laws for Reward Model Overoptimization
  • T-REG: Preference Optimization with Token-Level Reward Regularization cites Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  • T-REG: Preference Optimization with Token-Level Reward Regularization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Selective Preference Optimization via Token-Level Reward Function Estimation cites Scaling Laws for Reward Model Overoptimization
  • Selective Preference Optimization via Token-Level Reward Function Estimation cites Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  • Curiosity-Driven Reinforcement Learning from Human Feedback cites Fine-Grained Human Feedback Gives Better Rewards for Language Model Training
  • Curiosity-Driven Reinforcement Learning from Human Feedback cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Curiosity-Driven Reinforcement Learning from Human Feedback cites Training language models to follow instructions with human feedback
  • From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring cites T-REG: Preference Optimization with Token-Level Reward Regularization
  • From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoring cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers19

  1. From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content MonitoringYang Li, Qiang Sheng, Yehan Yang, Xueyao Zhang et al.NeurIPS 2025 · 21 citations
  2. Selective Preference Optimization via Token-Level Reward Function EstimationKailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang et al.EMNLP 2025 · 18 citations
  3. Curiosity-Driven Reinforcement Learning from Human FeedbackHaoran Sun, Yekun Chai, Shuohuan Wang, Yu Sun et al.ACL 2025 · 17 citations
  4. T-REG: Preference Optimization with Token-Level Reward RegularizationWenxuan Zhou, Shujian Zhang, Lingxiao Zhao, Tao MengACL 2025 · 11 citations
  5. Decomposed On-Policy Distillation for Vision-Language Reasoning: Steering Gradients for Visual GroundingHee Suk Yoon, Eunseop Yoon, Jaehyun Jang, SooHwan Eom et al.ICML 2026 · 6 citations

Builds on15

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Scaling Laws for Reward Model OveroptimizationLeo Gao, John Schulman, Jacob HiltonICML 2023 · 963 citations
  4. Fine-Grained Human Feedback Gives Better Rewards for Language Model TrainingZeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri et al.NeurIPS 2023 · 516 citations
  5. Prompting Large Language Model for Machine Translation: A Case StudyBiao Zhang, Barry Haddow, Alexandra BirchICML 2023 · 402 citations

Related papers

  1. RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward RedistributionJiahui Li, Lin Li, Tai-Wei Chang, Kun Kuang et al.EMNLP 2025
  2. Reward Modeling Requires Automatic Adjustment Based on Data QualityBinghai Wang, Rui Zheng, Lu Chen, Zhiheng Xi et al.EMNLP Findings 2024 · 2 citations
  3. Dense Reward for Free in Reinforcement Learning from Human FeedbackAlex James Chan, Hao Sun, Samuel Holt, Mihaela van der SchaarICML 2024 · 74 citations
  4. Discriminative Policy Optimization for Token-Level Reward ModelsHongzhan Chen, Tao Yang, Shiping Gao, Ruijun Chen et al.ICML 2025
  5. ARF-RLHF: Adaptive Reward-Following for RLHF through Emotion-Driven Self-Supervision and Trace-Biased Dynamic OptimizationYuxuan ZhangACL 2026
Dusk over the sea between two cliffs drawn in fine vertical lines