EMNLP Findings2025
Token Preference Optimization with Self-Calibrated Visual-Anchored Rewards for Hallucination Mitigation
Jihao Gu, Yingyao Wang, Meng Cao, Pi Bu, Jun Song, Bo Zheng, Yancheng He, Shilong Li
Abstract
Direct Preference Optimization (DPO) has been demonstrated to be highly effective in mitigating hallucinations in Large Vision Language Models (LVLMs) by aligning their outputs more closely with human preferences. Despite the recent progress, existing methods suffer from two drawbacks: 1) Lack of scalable token-level rewards; and 2) Neglect of visual-anchored tokens. To this end, we propose a novel Token Preference Optimization model with self-calibrated rewards (dubbed as TPO), which adaptively attends to visualcorrelated tokens without fine-grained annotations. Specifically, we introduce a tokenlevel visual-anchored reward as the difference of the logistic distributions of generated tokens conditioned on the raw image and the corrupted one. In addition, to highlight the informative visual-anchored tokens, a visualaware training objective is proposed to enhance more accurate token-level optimization. Extensive experimental results have manifested the state-of-the-art performance of the proposed TPO. For example, by building on top of LLaVA and Qwen, our TPO boosts the performance absolute improvement for hallucination benchmarks. Our code is available at https://github.com/alibaba/TPO.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
5 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Visual Instruction Tuning
- MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities cites Learning Transferable Visual Models From Natural Language Supervision
- Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs cites Visual Instruction Tuning
- Visual Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
- Cat-PO: Cross-modal Adaptive Token-rewards for Preference Optimization in Truthful Multimodal LLMs cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Cat-PO: Cross-modal Adaptive Token-rewards for Preference Optimization in Truthful Multimodal LLMs cites Visual Instruction Tuning
- Cat-PO: Cross-modal Adaptive Token-rewards for Preference Optimization in Truthful Multimodal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- DAMA: Data- and Model-aware Alignment of Multi-modal LLMs cites MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
- DAMA: Data- and Model-aware Alignment of Multi-modal LLMs cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- DAMA: Data- and Model-aware Alignment of Multi-modal LLMs cites Visual Instruction Tuning
- DAMA: Data- and Model-aware Alignment of Multi-modal LLMs cites Learning Transferable Visual Models From Natural Language Supervision
- DAMA: Data- and Model-aware Alignment of Multi-modal LLMs cites Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning cites Visual Instruction Tuning
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning cites Learning Transferable Visual Models From Natural Language Supervision
Cited by top-tier papers5
- Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMsXudong Li, Mengdan Zhang, Peixian Chen, Xiawu Zheng et al.NeurIPS 2025 · 4 citations
- Controllable Sequence Editing for Biological and Clinical TrajectoriesMichelle M. Li, Kevin Li, Yasha Ektefaie, Ying Jin et al.ICLR 2026
- Cat-PO: Cross-modal Adaptive Token-rewards for Preference Optimization in Truthful Multimodal LLMsZhixiao Zheng, Zheren Fu, Zhiyuan Yao, Dongming Zhang et al.ICLR 2026
- Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language ModelsHuatian Zhang, Zhendong Mao, Lei Zhang, Yongdong ZhangCVPR 2026
- DAMA: Data- and Model-aware Alignment of Multi-modal LLMsJinda Lu, Junkang Wu, Jinghan Li, Xiaojun Jia et al.ICML 2025
Builds on20
- Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- MM-Vet: Evaluating Large Multimodal Models for Integrated CapabilitiesWeihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang et al.ICML 2024 · 1,191 citations
- Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction TuningFuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang et al.ICLR 2024 · 476 citations
Related papers
- Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the KeyZhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu et al.CVPR 2025
- mDPO: Conditional Preference Optimization for Multimodal Large Language ModelsFei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu et al.EMNLP 2024 · 11 citations
- V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference OptimizationYuxi Xie, Guanzhen Li, Xiao Xu, Min-Yen KanEMNLP Findings 2024 · 20 citations
- Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMsByeonggeuk Lim, JungMin Yun, Junehyoung Kwon, Kyeonghyun Kim et al.ACL Findings 2026
- Calibrated Self-Rewarding Vision Language ModelsYiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang et al.NeurIPS 2024 · 77 citations
