EMNLP Findings2023
Coverage-based Example Selection for In-Context Learning
Shivanshu Gupta, Matt Gardner, Sameer Singh
Abstract
In-context learning (ICL), the ability of large language models to perform novel tasks by conditioning on a prompt with a few task examples, requires these examples to be informative about the test instance. The standard approach of independently ranking and selecting the most similar examples selects redundant examples while omitting important information. In this work, we show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects, e.g. reasoning patterns, of the test input. We further extend BSR and many standard metrics to easily optimizable set-level metrics, giving still better coverage of those salient aspects. On 15 datasets spanning 6 tasks and with 7 diverse LLMs, we show that (1) BSR is the superior metric for in-context example selection across the board, and (2) for compositional tasks, set selection using Set-BSR outperforms independent ranking by up to 17 points on average and, despite being trainingfree, surpasses methods that leverage task or LLM-specific training. 1 1 https://github.com/Shivanshu-Gupta/ icl-coverage Q: Add a meeting with Jim tomorrow. A: CreateEvent(AND(with_attendee(" Jim "),starts_at(Tomorrow()))) Q: Add an appointment with Jim for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),starts_at(Tomorrow()))) Q: Add a meeting with Jim and his manager for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),with_attendee(" his manager "),starts_at(Tomorrow()))) Q: Schedule a meeting with Doug and his boss for next week to review A: CreateEvent(AND(with_attendee(" Doug "), with_attendee(FindManager(" Doug ")),has_subject(" review "),starts_at(NextWeekList()))) Q: Add an appointment with Jim for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),starts_at(Tomorrow()))) Q: Add a meeting with Jim and his manager for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),with_attendee( FindManager(" Jim ")),starts_at(Tomorrow()))) References Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. Incontext examples selection for machine translation.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
21 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- Learning to Reason via Program Generation, Emulation, and Search cites Compositional Exemplars for In-context Learning
- On the Noise Robustness of In-Context Learning for Text Generation cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- On the Noise Robustness of In-Context Learning for Text Generation cites Compositional Exemplars for In-context Learning
- Using Natural Language Explanations to Improve Robustness of In-context Learning cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Using Natural Language Explanations to Improve Robustness of In-context Learning cites Compositional Exemplars for In-context Learning
- Using Natural Language Explanations to Improve Robustness of In-context Learning cites BERTScore: Evaluating Text Generation with BERT
- Using Natural Language Explanations to Improve Robustness of In-context Learning cites Calibrate Before Use: Improving Few-shot Performance of Language Models
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity cites Calibrate Before Use: Improving Few-shot Performance of Language Models
- Compositional Exemplars for In-context Learning cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks cites Compositional Exemplars for In-context Learning
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks cites Calibrate Before Use: Improving Few-shot Performance of Language Models
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars cites Compositional Exemplars for In-context Learning
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars cites Calibrate Before Use: Improving Few-shot Performance of Language Models
Cited by top-tier papers21
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of ExemplarsZhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu et al.NeurIPS 2024 · 44 citations
- On the Noise Robustness of In-Context Learning for Text GenerationHongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu et al.NeurIPS 2024 · 20 citations
- Learning to Reason via Program Generation, Emulation, and SearchNathaniel Weir, Muhammad Khalifa, Linlu Qiu, Orion Weller et al.NeurIPS 2024 · 16 citations
- GistScore: Learning Better Representations for In-Context Example Selection with Gist BottlenecksShivanshu Gupta, Clemens Rosenbaum, Ethan R. ElenbergICML 2024 · 10 citations
- Using Natural Language Explanations to Improve Robustness of In-context LearningXuanli He, Yuxiang Wu, Oana-Maria Camburu, Pasquale Minervini et al.ACL 2024 · 7 citations
Builds on10
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger et al.ICLR 2020 · 8,443 citations
- Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
- Measuring Compositional Generalization: A Comprehensive Method on Realistic DataDaniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman et al.ICLR 2020 · 401 citations
- Compositional Exemplars for In-context LearningJiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu et al.ICML 2023 · 188 citations
Related papers
- SCOI: Syntax-augmented Coverage-based In-context Example Selection for Machine TranslationChenming Tang, Zhixiang Wang, Yunfang WuEMNLP 2024 · 1 citation
- What Do Language Models Learn in Context? The Structured Task HypothesisJiaoda Li, Yifan Hou, Mrinmaya Sachan, Ryan CotterellACL 2024 · 5 citations
- CTQScorer: Combining Multiple Features for In-context Example Selection for Machine TranslationAswanth M., Ratish Puduppully, Raj Dabre, Anoop KunchukuttanEMNLP Findings 2023 · 4 citations
- Effective In-Context Example Selection through Data CompressionZhongxiang Sun, Kepu Zhang, Haoyu Wang, Xiao Zhang et al.ACL Findings 2024
- Data Curation Alone Can Stabilize In-context LearningTing-Yun Chang, Robin JiaACL 2023 · 11 citations
