EMNLP Findings2023
MEAL: Stable and Active Learning for Few-Shot Prompting
Abdullatif Köksal, Timo Schick, Hinrich Schütze
Abstract
Few-shot classification has made great strides due to foundation models that, through priming and prompting, are highly effective few-shot learners. However, this approach has high variance both across different sets of few shots (data selection) and across different finetuning runs (run variability). This is problematic not only because it impedes the fair comparison of different approaches, but especially because it makes few-shot learning too unreliable for many real-world applications. To alleviate these issues, we make two contributions for more stable and effective few-shot learning: First, we propose novel ensembling methods and show that they substantially reduce run variability. Second, we introduce a new active learning (AL) criterion for data selection and present the first AL-based approach specifically tailored towards prompt-based learning.
In our experiments, we show that our combined method, MEAL (Multiprompt finetuning and prediction Ensembling with Active Learning), improves overall performance of prompt-based finetuning by 2.3 points on five diverse tasks. We publicly share our code and data splits in https://github.com/akoksal/MEAL.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
6 top-tier papers have cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- The Good, the Bad, and the Debatable: A Survey on the Impacts of Data for In-Context Learning cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentation cites On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
- Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance cites Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
- Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance cites On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
- Active Learning for BERT: An Empirical Study cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
- Active Learning for BERT: An Empirical Study cites Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines cites ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
Cited by top-tier papers6
- Active Prompting with Chain-of-Thought for Large Language ModelsShizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan et al.ACL 2024 · 40 citations
- Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even PerformanceBranislav Pecher, Ivan Srba, Mária BielikováEMNLP 2025 · 6 citations
- Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentationJán Cegin, Branislav Pecher, Jakub Simko, Ivan Srba et al.ACL 2024 · 5 citations
- Flexible and Adaptable Summarization via Expertise SeparationXiuying Chen, Mingzhe Li, Shen Gao, Xin Cheng et al.SIGIR 2024 · 2 citations
- The Good, the Bad, and the Debatable: A Survey on the Impacts of Data for In-Context LearningStephanie Schoch, Yangfeng JiEMNLP 2025
Builds on10
- ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
- Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order SensitivityYao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel et al.ACL 2022 · 1,494 citations
- Deep Batch Active Learning by Diverse, Uncertain Gradient Lower BoundsJordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford et al.ICLR 2020 · 974 citations
- On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong BaselinesMarius Mosbach, Maksym Andriushchenko, Dietrich KlakowICLR 2021 · 448 citations
- Active Learning for BERT: An Empirical StudyLiat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch et al.EMNLP 2020 · 144 citations
Related papers
- FSL-QuickBoost: Minimal-Cost Ensemble for Few-Shot LearningYunwei Bai, Bill Yang Cai, Ying Kiat Tan, Zangwei Zheng et al.ACM MM 2024
- Prompt-Based Meta-Learning For Few-shot Text ClassificationHaoxing Zhang, Xiaofeng Zhang, Haibo Huang, Lei YuEMNLP 2022 · 34 citations
- Automated Few-Shot Classification with Instruction-Finetuned Language ModelsRami Aly, Xingjian Shi, Kaixiang Lin, Aston Zhang et al.EMNLP Findings 2023 · 3 citations
- Cold-Start Data Selection for Better Few-shot Language Model Fine-tuning: A Prompt-based Uncertainty Propagation ApproachYue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang et al.ACL 2023 · 12 citations
- Revisiting Self-training for Few-shot Learning of Language ModelYiming Chen, Yan Zhang, Chen Zhang, Grandee Lee et al.EMNLP 2021 · 35 citations
