Lune

EMNLP Findings2024

BAPO: Base-Anchored Preference Optimization for Overcoming Forgetting in Large Language Models Personalization

Gihun Lee, Minchan Jeong, Yujin Kim, Hojung Jung, Jaehoon Oh, SangMook Kim, Se-Young Yun

Abstract

While learning to align Large Language Models (LLMs) with human preferences has shown remarkable success, aligning these models to meet the diverse user preferences presents further challenges in preserving previous knowledge. This paper examines the impact of personalized preference optimization on LLMs, revealing that the extent of knowledge loss varies significantly with preference heterogeneity. Although previous approaches have utilized the KL constraint between the reference model and the policy model, we observe that they fail to maintain global knowledge and general alignment when facing personalized preferences. To this end, we introduce Base-Anchored Preference Optimization (BAPO), a simple yet effective approach that utilizes the initial responses of reference model to mitigate forgetting while accommodating personalized alignment. BAPO effectively adapts to diverse user preferences while minimally affecting global knowledge or general alignment.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Measuring Massive Multitask Language Understanding cites PIQA: Reasoning about Physical Commonsense in Natural Language
  • Aligning VLM Assistants with Personalized Situated Cognition cites LoRA: Low-Rank Adaptation of Large Language Models
  • Aligning VLM Assistants with Personalized Situated Cognition cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  • QLoRA: Efficient Finetuning of Quantized LLMs cites LoRA: Low-Rank Adaptation of Large Language Models
  • QLoRA: Efficient Finetuning of Quantized LLMs cites Measuring Massive Multitask Language Understanding
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Aligning VLM Assistants with Personalized Situated CognitionYongqi Li, Shen Zhou, Xiaohu Li, Xin Miao et al.ACL 2025

Builds on21

  1. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  2. Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
  3. Measuring Massive Multitask Language UnderstandingDan Hendrycks, Collin Burns, Steven Basart, Andy Zou et al.ICLR 2021 · 7,905 citations
  4. QLoRA: Efficient Finetuning of Quantized LLMsTim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke ZettlemoyerNeurIPS 2023 · 5,863 citations
  5. PIQA: Reasoning about Physical Commonsense in Natural LanguageYonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao et al.AAAI 2020 · 2,916 citations

Related papers

  1. AMoPO: Adaptive Multi-objective Preference Optimization without Reward Models and Reference ModelsQi Liu, Jingqing Ruan, Hao Li, Haodong Zhao et al.ACL Findings 2025 · 2 citations
  2. Cal-DPO: Calibrated Direct Preference Optimization for Language Model AlignmentTeng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li et al.NeurIPS 2024 · 76 citations
  3. Knowledge Editing in Language Models via Adapted Direct Preference OptimizationAmit Rozner, Barak Battash, Lior Wolf, Ofir LindenbaumEMNLP Findings 2024 · 1 citation
  4. COPR: Continual Human Preference Learning via Optimal Policy RegularizationHan Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai et al.ACL Findings 2025 · 3 citations
  5. Token-level Direct Preference OptimizationYongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang et al.ICML 2024 · 136 citations
Dusk over the sea between two cliffs drawn in fine vertical lines