Lune

ACL2024Top-tier venue

Revisiting Demonstration Selection Strategies in In-Context Learning

Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, Dacheng Tao

Abstract

Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL), where a few examples are used to describe a task to the model. However, the performance of ICL varies significantly with the choice of demonstrations, and previous research usually focuses on the data aspect ignoring the model's effect. In this work, we first revisit the factors contributing to this variance from the model aspect, and find that the demonstration choice is both data-and model-dependent. We further propose a conjecture that the performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples, and accordingly propose a dataand model-dependent demonstration selection method, TopK + ConE. Empirically, our method yields consistent improvements in both language understanding and generation tasks with different model scales. Further analyses confirm that, besides the generality and stability under different circumstances, our method provides a unified explanation for the effectiveness of previous methods. Code is publicly available at https://github.com/Romainpkq/ revisit_demon_selection_in_ICL .

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

19 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Topic Coverage-based Demonstration Retrieval for In-Context Learning cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
  • Topic Coverage-based Demonstration Retrieval for In-Context Learning cites Language Models are Few-Shot Learners
  • On the Noise Robustness of In-Context Learning for Text Generation cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
  • On the Noise Robustness of In-Context Learning for Text Generation cites Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
  • On the Noise Robustness of In-Context Learning for Text Generation cites Language Models are Few-Shot Learners
  • Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Language Models are Few-Shot Learners
  • Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Calibrate Before Use: Improving Few-shot Performance of Language Models
  • From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When cites Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
  • From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When cites Language Models are Few-Shot Learners
  • What Makes a Good Natural Language Prompt? cites Training language models to follow instructions with human feedback
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Language Models are Few-Shot Learners
  • Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? cites Calibrate Before Use: Improving Few-shot Performance of Language Models
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
  • TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data Consistency cites On the Noise Robustness of In-Context Learning for Text Generation
  • Calibrate Before Use: Improving Few-shot Performance of Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers19

  1. On the Noise Robustness of In-Context Learning for Text GenerationHongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu et al.NeurIPS 2024 · 20 citations
  2. What Makes a Good Natural Language Prompt?Do Xuan Long, Duy Dinh, Ngoc-Hai Nguyen, Kenji Kawaguchi et al.ACL 2025 · 13 citations
  3. From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and WhenKevin Christian Wibisono, Yixin WangNeurIPS 2024 · 5 citations
  4. Topic Coverage-based Demonstration Retrieval for In-Context LearningWonbin Kweon, SeongKu Kang, Runchu Tian, Pengcheng Jiang et al.EMNLP 2025 · 4 citations
  5. TestNUC: Enhancing Test-Time Computing Approaches and Scaling through Neighboring Unlabeled Data ConsistencyHenry Peng Zou, Zhengyao Gu, Yue Zhou, Yankai Chen et al.ACL 2025 · 3 citations

Builds on10

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
  4. Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
  5. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe et al.EMNLP 2022 · 634 citations

Related papers

  1. Active Example Selection for In-Context LearningYiming Zhang, Shi Feng, Chenhao TanEMNLP 2022 · 84 citations
  2. DICE: Dynamic In-Context Example Selection in LLM Agents via Efficient Knowledge TransferRuoyu Wang, Junda Wu, Yu Xia, Tong Yu et al.KDD 2026 · 6 citations
  3. What Do Language Models Learn in Context? The Structured Task HypothesisJiaoda Li, Yifan Hou, Mrinmaya Sachan, Ryan CotterellACL 2024 · 5 citations
  4. Learning to Select In-Context Demonstration Preferred by Large Language ModelZheng Zhang, Shaocheng Lan, Lei Song, Jiang Bian et al.ACL Findings 2025 · 5 citations
  5. How Many Demonstrations Do You Need for In-context Learning?Jiuhai Chen, Lichang Chen, Chen Zhu, Tianyi ZhouEMNLP Findings 2023 · 12 citations
Dusk over the sea between two cliffs drawn in fine vertical lines