ACL Findings2026
MEAV: Model Editing with Alignment Vectors for inference time LLM alignment in single and multidomain preference spectrum
Sadat Shahriar, Zheng Qi, Nikolaos Pappas, Srikanth Doss, Kishaloy Halder, Monica Sunkara, Manuel Mager, Yassine Benajiba
Abstract
Aligning Large Language Models (LLM) to address subjectivity and nuanced preference levels requires adequate flexibility and control, which can be a resource-intensive and time-consuming procedure.Existing trainingtime alignment methods require full re-training when a change is needed and inference-time ones typically require access to the reward model at each inference step.We introduce MEAV, an inference-time model-editingbased LLM alignment method that learns encoded representations of preference dimensions, called Alignment Vectors (AV).These representations enable dynamic adjusting of the model behavior during inference through simple linear operations.Here, we focus on three gradual response levels across three specialized domains: medical, legal, and financial, exemplifying its practical potential.This new alignment paradigm introduces adjustable preference knobs during inference, allowing users to tailor their LLM outputs while reducing the inference cost by half compared to the prompt engineering approach.Additionally, we find that AVs are transferable across different finetuning stages of the same model, demonstrating their flexibility.AVs also facilitate multidomain, diverse preference alignment, making the process 12x faster than the retraining approach.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment cites Learning to summarize with human feedback
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Builds on11
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Inference-Time Intervention: Eliciting Truthful Answers from a Language ModelKenneth Li, Oam Patel, Fernanda B. Viégas, Hanspeter Pfister et al.NeurIPS 2023 · 1,549 citations
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free LunchLe Yu, Bowen Yu, Haiyang Yu, Fei Huang et al.ICML 2024 · 605 citations
- Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference AdjustmentRui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu et al.ICML 2024 · 144 citations
- Learning to summarize with human feedbackNisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler et al.NeurIPS 2020 · 124 citations
Related papers
- REAR: Test-time Preference Realignment through Reward DecompositionFuxiang Zhang, Pengcheng Wang, Chenran Li, Yi-Chen Li et al.ICML 2026
- Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward ModelsChenglong Wang, Yifu Huo, Yang Gan, Yongyu Mu et al.AAAI 2026 · 1 citation
- MetaAligner: Towards Generalizable Multi-Objective Alignment of Language ModelsKailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang et al.NeurIPS 2024 · 49 citations
- Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual FeedbackYafu Li, Xuyang Hu, Xiaoye Qu, Linjie Li et al.ICML 2025
- Data Selection for LLM Alignment Using Fine-Grained PreferencesJia Zhang, Yao Liu, Chen-Xi Zhang, Yi Liu et al.ICLR 2026 · 1 citation
