SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization
Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang
Abstract
As language models continue to scale, Large Language Models (LLMs) have exhibited emerging capabilities in In-Context Learning (ICL), enabling them to solve language tasks by prefixing a few in-context demonstrations (ICDs) as context. Inspired by these advancements, researchers have extended these techniques to develop Large Multimodal Models (LMMs) with ICL capabilities. However, existing LMMs face a critical issue: they often fail to effectively leverage the visual context in multimodal demonstrations and instead simply follow textual patterns. This indicates that LMMs do not achieve effective alignment between multimodal demonstrations and model outputs. To address this problem, we propose Symbol Demonstration Direct Preference Optimization (SymDPO). Specifically, SymDPO aims to break the traditional paradigm of constructing multimodal demonstrations by using random symbols to replace text answers within instances. This forces the model to carefully understand the demonstration images and establish a relationship between the images and the symbols to answer questions correctly. We validate the effectiveness of this method on multiple benchmarks, demonstrating that with SymDPO, LMMs can more effectively understand the multimodal context within examples and utilize this knowledge to answer questions better. Code is available at https://github.com/APiaoG/SymDPO .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
7 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive Images cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning cites Visual Instruction Tuning
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning cites Language Models are Few-Shot Learners
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context Learning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization cites Direct Preference Optimization: Your Language Model is Secretly a Reward Model
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization cites Visual Instruction Tuning
- Agentic Design Review System cites Visual Instruction Tuning
- Agentic Design Review System cites Flamingo: a Visual Language Model for Few-Shot Learning
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model cites Language Models are Few-Shot Learners
- Visual Instruction Tuning cites Language Models are Few-Shot Learners
- Visual Instruction Tuning cites Flamingo: a Visual Language Model for Few-Shot Learning
- Flamingo: a Visual Language Model for Few-Shot Learning cites Language Models are Few-Shot Learners
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models cites Visual Instruction Tuning
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models cites Language Models are Few-Shot Learners
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models cites Flamingo: a Visual Language Model for Few-Shot Learning
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning cites Language Models are Few-Shot Learners
Cited by top-tier papers7
- Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference OptimizationWenqi Liu, Xuemeng Song, Jiaxi Li, Yinwei Wei et al.NeurIPS 2025 · 18 citations
- Symmetrical Visual Contrastive Optimization: Aligning Vision-Language Models with Minimal Contrastive ImagesShengguang Wu, Fan-Yun Sun, Kaiyue Wen, Nick HaberACL 2025 · 12 citations
- Make LVLMs Focus: Context-Aware Attention Modulation for Better Multimodal In-Context LearningYanshu Li, Jianjiang Yang, Ziteng Yang, Bozheng Li et al.AAAI 2026 · 9 citations
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language ModelsHengzhuang Li, Xinsong Zhang, QIMING PENG, Bin Luo et al.CVPR 2026 · 2 citations
- Agentic Design Review SystemSayan Nag, K. J. Joseph, Koustava Goswami, Vlad I. Morariu et al.AAAI 2026 · 1 citation
Builds on20
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- Visual Instruction TuningHaotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae LeeNeurIPS 2023 · 11,349 citations
- Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning et al.NeurIPS 2023 · 10,924 citations
- Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context LearningHaokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta et al.NeurIPS 2022 · 1,483 citations
Related papers
- mDPO: Conditional Preference Optimization for Multimodal Large Language ModelsFei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu et al.EMNLP 2024 · 11 citations
- Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference OptimizationYuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang et al.ACL Findings 2025 · 6 citations
- Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMsXudong Li, Mengdan Zhang, Peixian Chen, Xiawu Zheng et al.NeurIPS 2025 · 4 citations
- Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context LearningCheng Chen, Yunpeng Zhai, Yifan Zhao, Jinyang Gao et al.CVPR 2025
- LIVE: Learnable In-Context Vector for Visual Question AnsweringYingzhe Peng, Chenduo Hao, Xinting Hu, Jiawei Peng et al.NeurIPS 2024
