Comparing Comparisons: Informative and Easy Human Feedback with Distinguishability Queries
Xuening Feng, Zhaohui Jiang, Timo Kaufmann, Eyke Hüllermeier, Paul Weng, Yifei Zhu
Abstract
Learning human objectives from preference feedback has significantly advanced reinforcement learning (RL) in domains where objectives are hard to formalize. However, traditional methods based on pairwise trajectory comparisons face notable challenges, including the difficulty in comparing trajectories with subtle differences and the limitation of conveying only ordinal information, limiting direct inference of preference strength. In this paper, we introduce a novel distinguishability query, enabling humans to express preference strength by comparing two pairs of trajectories. Labelers first indicate which of two pairs is easier to distinguish, then provide preference feedback only on the easier pair. Our proposed query type directly captures preference strength and is expected to reduce the cognitive load on the labeler. We further connect this query to cardinal utility and difference relations and develop an efficient query selection scheme to achieve a better tradeoff between query informativeness and easiness. Experimental results demonstrate the potential of our method for faster, data-efficient learning and improved user-friendliness in RLHF benchmarks, particularly in classical control settings where preference strength is critical for expected utility maximization.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
3 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning cites PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
- Progressive Learning with Human Feedback for Personalized Adaptive Video Streaming cites PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
- PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models cites SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learning
- PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models cites PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength Learning cites Training language models to follow instructions with human feedback
Cited by top-tier papers3
- PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation ModelsRuiqi Wang, Dezhong Zhao, Ziqin Yuan, Tianyu Shao et al.NeurIPS 2025 · 9 citations
- ResponseRank: Data-Efficient Reward Modeling through Preference Strength LearningTimo Kaufmann, Yannick Metz, Daniel A. Keim, Eyke HüllermeierNeurIPS 2025 · 2 citations
- Progressive Learning with Human Feedback for Personalized Adaptive Video StreamingZhaohui Jiang, Xuening Feng, Tianchi Huang, Ruixiao Zhang et al.ACM MM 2025
Builds on10
- Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
- PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingKimin Lee, Laura M. Smith, Pieter AbbeelICML 2021 · 380 citations
- Reward-rational (implicit) choice: A unifying formalism for reward learningHong Jun Jeon, Smitha Milli, Anca D. DraganNeurIPS 2020 · 219 citations
- SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement LearningJongjin Park, Younggyo Seo, Jinwoo Shin, Honglak Lee et al.ICLR 2022 · 116 citations
- Preference-based Reinforcement Learning with Finite-Time GuaranteesYichong Xu, Ruosong Wang, Lin F. Yang, Aarti Singh et al.NeurIPS 2020 · 82 citations
Related papers
- CLARIFY: Contrastive Preference Reinforcement Learning for Untangling Ambiguous QueriesNi Mu, Hao Hu, Xiao Hu, Yiqin Yang et al.ICML 2025
- DUO: Diverse, Uncertain, On-Policy Query Generation and Selection for Reinforcement Learning from Human FeedbackXuening Feng, Zhaohui Jiang, Timo Kaufmann, Puchen Xu et al.AAAI 2025 · 7 citations
- Efficient Preference-Based Reinforcement Learning: Randomized Exploration meets Experimental DesignAndreas Schlaginhaufen, Reda Ouhamma, Maryam KamgarpourNeurIPS 2025 · 4 citations
- Sequential Preference Ranking for Efficient Reinforcement Learning from Human FeedbackMinyoung Hwang, Gunmin Lee, Hogun Kee, Chanwoo Kim et al.NeurIPS 2023 · 24 citations
- Towards Theoretical Understanding of Sequential Decision Making with Preference FeedbackSimone Drago, Marco Mussi, Alberto Maria MetelliICML 2025
