EMNLP Findings2023
Coverage-based Example Selection for In-Context Learning
Shivanshu Gupta, Matt Gardner, Sameer Singh
摘要
In-context learning (ICL), the ability of large language models to perform novel tasks by conditioning on a prompt with a few task examples, requires these examples to be informative about the test instance. The standard approach of independently ranking and selecting the most similar examples selects redundant examples while omitting important information. In this work, we show that BERTScore-Recall (BSR) selects better examples that demonstrate more of the salient aspects, e.g. reasoning patterns, of the test input. We further extend BSR and many standard metrics to easily optimizable set-level metrics, giving still better coverage of those salient aspects. On 15 datasets spanning 6 tasks and with 7 diverse LLMs, we show that (1) BSR is the superior metric for in-context example selection across the board, and (2) for compositional tasks, set selection using Set-BSR outperforms independent ranking by up to 17 points on average and, despite being trainingfree, surpasses methods that leverage task or LLM-specific training. 1 1 https://github.com/Shivanshu-Gupta/ icl-coverage Q: Add a meeting with Jim tomorrow. A: CreateEvent(AND(with_attendee(" Jim "),starts_at(Tomorrow()))) Q: Add an appointment with Jim for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),starts_at(Tomorrow()))) Q: Add a meeting with Jim and his manager for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),with_attendee(" his manager "),starts_at(Tomorrow()))) Q: Schedule a meeting with Doug and his boss for next week to review A: CreateEvent(AND(with_attendee(" Doug "), with_attendee(FindManager(" Doug ")),has_subject(" review "),starts_at(NextWeekList()))) Q: Add an appointment with Jim for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),starts_at(Tomorrow()))) Q: Add a meeting with Jim and his manager for tomorrow A: CreateEvent(AND(with_attendee(" Jim "),with_attendee( FindManager(" Jim ")),starts_at(Tomorrow()))) References Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. Incontext examples selection for machine translation.
问 Lune
几分钟读透这篇 Paper,不用再花一下午。
Lune 能读完全文,也能读引用它的顶会 Paper,讲清它的结论、证据和局限,每一点都注明出自哪篇 Paper。不用硬啃,也不靠猜。
已有 21 篇顶会 Paper 引用了它。别等审稿人问起,先弄清它们发现了什么。
脉络
- Learning to Reason via Program Generation, Emulation, and Search 引用了 Compositional Exemplars for In-context Learning
- On the Noise Robustness of In-Context Learning for Text Generation 引用了 Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- On the Noise Robustness of In-Context Learning for Text Generation 引用了 Compositional Exemplars for In-context Learning
- Using Natural Language Explanations to Improve Robustness of In-context Learning 引用了 Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Using Natural Language Explanations to Improve Robustness of In-context Learning 引用了 Compositional Exemplars for In-context Learning
- Using Natural Language Explanations to Improve Robustness of In-context Learning 引用了 BERTScore: Evaluating Text Generation with BERT
- Using Natural Language Explanations to Improve Robustness of In-context Learning 引用了 Calibrate Before Use: Improving Few-shot Performance of Language Models
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity 引用了 Calibrate Before Use: Improving Few-shot Performance of Language Models
- Compositional Exemplars for In-context Learning 引用了 Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks 引用了 Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks 引用了 Compositional Exemplars for In-context Learning
- GistScore: Learning Better Representations for In-Context Example Selection with Gist Bottlenecks 引用了 Calibrate Before Use: Improving Few-shot Performance of Language Models
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars 引用了 Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars 引用了 Compositional Exemplars for In-context Learning
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of Exemplars 引用了 Calibrate Before Use: Improving Few-shot Performance of Language Models
引用它的顶会 Paper21
- Prompt Optimization with EASE? Efficient Ordering-aware Automated Selection of ExemplarsZhaoxuan Wu, Xiaoqiang Lin, Zhongxiang Dai, Wenyang Hu 等NeurIPS 2024 · 44 次引用
- On the Noise Robustness of In-Context Learning for Text GenerationHongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu 等NeurIPS 2024 · 20 次引用
- Learning to Reason via Program Generation, Emulation, and SearchNathaniel Weir, Muhammad Khalifa, Linlu Qiu, Orion Weller 等NeurIPS 2024 · 16 次引用
- GistScore: Learning Better Representations for In-Context Example Selection with Gist BottlenecksShivanshu Gupta, Clemens Rosenbaum, Ethan R. ElenbergICML 2024 · 10 次引用
- Using Natural Language Explanations to Improve Robustness of In-context LearningXuanli He, Yuxiang Wu, Oana-Maria Camburu, Pasquale Minervini 等ACL 2024 · 7 次引用
它引用的 Paper10
- BERTScore: Evaluating Text Generation with BERTTianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger 等ICLR 2020 · 8,443 次引用
- Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein 等ICML 2021 · 1,843 次引用
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel 等ACL 2022 · 1,494 次引用
- Measuring Compositional Generalization: A Comprehensive Method on Realistic DataDaniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman 等ICLR 2020 · 401 次引用
- Compositional Exemplars for In-context LearningJiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu 等ICML 2023 · 188 次引用
相关 Paper
- SCOI: Syntax-augmented Coverage-based In-context Example Selection for Machine TranslationChenming Tang, Zhixiang Wang, Yunfang WuEMNLP 2024 · 1 次引用
- What Do Language Models Learn in Context? The Structured Task HypothesisJiaoda Li, Yifan Hou, Mrinmaya Sachan, Ryan CotterellACL 2024 · 5 次引用
- CTQScorer: Combining Multiple Features for In-context Example Selection for Machine TranslationAswanth M., Ratish Puduppully, Raj Dabre, Anoop KunchukuttanEMNLP Findings 2023 · 4 次引用
- Effective In-Context Example Selection through Data CompressionZhongxiang Sun, Kepu Zhang, Haoyu Wang, Xiao Zhang 等ACL Findings 2024
- Data Curation Alone Can Stabilize In-context LearningTing-Yun Chang, Robin JiaACL 2023 · 11 次引用
