Lune

ICLR2025Top-tier venue

Anyprefer: An Agentic Framework for Preference Data Synthesis

Yiyang Zhou, Zhaoyang Wang, Tianle Wang, Shangyu Xing, Peng Xia, Bo Li, Kaiyuan Zheng, Zijian Zhang, Zhaorun Chen, Wenhao Zheng, Xuchao Zhang, Chetan Bansal

Abstract

High-quality preference data is essential for aligning foundation models with human values through preference learning. However, manual annotation of such data is often time-consuming and costly. Recent methods often adopt a self-rewarding approach, where the target model generates and annotates its own preference data, but this can lead to inaccuracies since the reward model shares weights with the target model, thereby amplifying inherent biases. To address these issues, we propose Anyprefer, a framework designed to synthesize high-quality preference data for aligning the target model. Anyprefer frames the data synthesis process as a cooperative two-player Markov Game, where the target model and the judge model collaborate together. Here, a series of external tools are introduced to assist the judge model in accurately rewarding the target model's responses, mitigating biases in the rewarding process. In addition, a feedback mechanism is introduced to optimize prompts for both models, enhancing collaboration and improving data quality. The synthesized data is compiled into a new preference dataset, Anyprefer-V1, consisting of 58K high-quality preference pairs. Extensive experiments show that Anyprefer significantly improves model alignment performance across four main applications, covering 21 datasets, achieving average improvements of 18.55% in five natural language generation datasets, 3.66% in nine vision-language understanding datasets, 30.05% in three medical image analysis datasets, and 16.00% in four visuo-motor control tasks.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

5 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization cites Self-Rewarding Language Models
  • MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
  • MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Training language models to follow instructions with human feedback
  • Beyond Surface Features: Advancing Medical Vision-Language Alignment via Dynamic Evidence-Guided Preference Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Beyond Surface Features: Advancing Medical Vision-Language Alignment via Dynamic Evidence-Guided Preference Optimization cites Self-Rewarding Language Models
  • Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Training language models to follow instructions with human feedback
  • Self-Rewarding Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • Self-Rewarding Language Models cites Training language models to follow instructions with human feedback
  • The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage cites Self-Rewarding Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers5

  1. Text-Printed Image: Bridging the Image-Text Modality Gap for Text-centric Training of Large Vision-Language ModelsShojiro Yamabe, Futa Waseda, Daiki Shiono, Tsubasa TakahashiCVPR 2026 · 2 citations
  2. MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference OptimizationKangyu Zhu, Peng Xia, Yun Li, Hongtu Zhu et al.ICML 2025
  3. Beyond Surface Features: Advancing Medical Vision-Language Alignment via Dynamic Evidence-Guided Preference OptimizationZixuan Huang, Zhihong Zhu, Xiaolong Liu, Yanchao Hao et al.ACL 2026
  4. SelPE: Progressive Selection for Private Structured Text SynthesisXuancheng Zhu, Guoshun Nan, Han Zhang, Ben Niu et al.KDD 2026
  5. The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera FootagePreni Golazizian, Elnaz Rahmati, Jackson Trager, Zhivar Sourati et al.ACL 2026

Builds on22

  1. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language ModelsJunnan Li, Dongxu Li, Silvio Savarese, Steven C. H. HoiICML 2023 · 7,873 citations
  4. MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesWeihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.ICML 2024 · 1,191 citations
  5. Self-Rewarding Language ModelsWeizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li et al.ICML 2024 · 569 citations

Related papers

  1. Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM GamePengyu Cheng, Yifan Yang, Jian Li, Yong Dai et al.ACL Findings 2024 · 6 citations
  2. Larger or Smaller Reward Margins to Select Preferences for LLM Alignment?Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang et al.ICML 2025
  3. Self-Steering Optimization: Autonomous Preference Optimization for Large Language ModelsHao Xiang, Bowen Yu, Hongyu Lin, Keming Lu et al.ACL Findings 2025
  4. Preference-Strength-Aware Self-Improving Alignment with Generative Preference ModelsYuanzhao Zhai, Zhuo Zhang, Cheng Yang, Kele Xu et al.SIGIR 2025
  5. Alignment Data Map for Efficient Preference Data Selection and DiagnosisSeohyeong Lee, Eunwon Kim, Hwaran Lee, Buru ChangACL Findings 2026
Dusk over the sea between two cliffs drawn in fine vertical lines