Lune

CVPR Findings2026

Learning to Select Visual In-Context Demonstrations

Eugene Lee, Yu-Chi Lin, Jiajie Diao

Abstract

Multimodal Large Language Models (MLLMs) adapt to visual tasks via in-context learning (ICL), which relies heavily on demonstration quality. The dominant demonstration selection strategy is unsupervised k-Nearest Neighbor (kNN) search. While simple, this similarity-first approach is suboptimal for complex factual regression tasks; it selects redundant examples that fail to capture the task's full output range. We reframe selection as a sequential decisionmaking problem and introduce Learning to Select Demonstrations (LSD), training a Reinforcement Learning agent to construct optimal demonstration sets. Using a Dueling DQN with a query-centric Transformer Decoder, our agent learns a policy that maximizes MLLM downstream performance. Evaluating across five visual regression benchmarks, we uncover a crucial dichotomy: while kNN remains optimal for subjective preference tasks, LSD significantly outperforms baselines on objective, factual regression tasks. By balancing visual relevance with diversity, LSD better defines regression boundaries, illuminating when learned selection is strictly necessary for visual ICL.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • A Survey on In-context Learning cites Compositional Exemplars for In-context Learning
  • A Survey on In-context Learning cites Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on23

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsJason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma et al.NeurIPS 2022 · 22,562 citations
  2. Sigmoid Loss for Language Image Pre-TrainingXiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, Lucas BeyerICCV 2023 · 2,932 citations
  3. A Survey on In-context LearningQingxiu Dong, Lei Li, Damai Dai, Ce Zheng et al.EMNLP 2024 · 479 citations
  4. What Makes Good Examples for Visual In-Context Learning?Yuanhan Zhang, Kaiyang Zhou, Ziwei LiuNeurIPS 2023 · 219 citations
  5. Compositional Exemplars for In-context LearningJiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu et al.ICML 2023 · 188 citations

Related papers

  1. Active Example Selection for In-Context LearningYiming Zhang, Shi Feng, Chenhao TanEMNLP 2022 · 84 citations
  2. Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context LearningCheng Chen, Yunpeng Zhai, Yifan Zhao, Jinyang Gao et al.CVPR 2025
  3. DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge TransferRuoyu Wang, Junda Wu, Yu Xia, Tong Yu et al.KDD 2026 · 6 citations
  4. Auto-regressive In-context Demonstration SelectionYunzhe Qi, Sirui Chen, Jiaru Zou, Jingrui HeICML 2026
  5. In-Context Compositional Generalization for Large Vision-Language ModelsChuanhao Li, Chenchen Jing, Zhen Li, Mingliang Zhai et al.EMNLP 2024 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines