ACL Findings2023
Revisiting Sample Size Determination in Natural Language Understanding
Ernie Chang, Muhammad Hassan Rashid, Pin-Jie Lin, Changsheng Zhao, Vera Demberg, Yangyang Shi, Vikas Chandra
Abstract
Knowing exactly how many data points need to be labeled to achieve a certain model performance is a hugely beneficial step towards reducing the overall budgets for annotation.It pertains to both active learning and traditional data annotation, and is particularly beneficial for low resource scenarios.Nevertheless, it remains a largely under-explored area of research in NLP.We therefore explored various techniques for estimating the training sample size necessary to achieve a targeted performance value.We derived a simple yet effective approach to predict the maximum achievable model performance based on small amount of training sampleswhich serves as an early indicator during data annotation for data quality and sample size determination.We performed ablation studies on four language understanding tasks, and showed that the proposed approach allows us to forecast model performance within a small margin of mean absolute error (∽ 0.9%) with only 10% data 1 .
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
Lineage
- Deep Learning on a Data Diet: Finding Important Examples Early in Training cites Selection via Proxy: Efficient Data Selection for Deep Learning
- Deep Learning on a Data Diet: Finding Important Examples Early in Training cites GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning
- RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning cites GLISTER: Generalization based Data Subset Selection for Efficient and Robust Learning
Builds on5
- Deep Learning on a Data Diet: Finding Important Examples Early in TrainingMansheej Paul, Surya Ganguli, Gintare Karolina DziugaiteNeurIPS 2021 · 806 citations
- Selection via Proxy: Efficient Data Selection for Deep LearningCody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman et al.ICLR 2020 · 462 citations
- GLISTER: Generalization based Data Subset Selection for Efficient and Robust LearningKrishnaTeja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, Rishabh K. IyerAAAI 2021 · 300 citations
- Cold-start Active Learning through Self-supervised Language ModelingMichelle Yuan, Hsuan-Tien Lin, Jordan L. Boyd-GraberEMNLP 2020 · 128 citations
- RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised LearningKrishnaTeja Killamsetty, Xujiang Zhao, Feng Chen, Rishabh K. IyerNeurIPS 2021 · 115 citations
Related papers
- Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even PerformanceBranislav Pecher, Ivan Srba, Mária BielikováEMNLP 2025 · 6 citations
- Want To Reduce Labeling Cost? GPT-3 Can HelpShuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu et al.EMNLP Findings 2021 · 177 citations
- Pre-train or Annotate? Domain Adaptation with a Constrained BudgetFan Bai, Alan Ritter, Wei XuEMNLP 2021 · 25 citations
- DA-Pred: Performance Prediction for Text Summarization under Domain-Shift and Instruct-TuningAnum Afzal, Florian Matthes, Alexander R. FabbriEMNLP 2025
- Scaling Up Active Testing to Large Language ModelsGabrielle Berrada, Jannik Kossen, Freddie Bickford Smith, Muhammed Razzak et al.NeurIPS 2025 · 11 citations
