ACL Findings2022
EICO: Improving Few-Shot Text Classification via Explicit and Implicit Consistency Regularization
Lei Zhao, Cheng Yao
Abstract
While the prompt-based fine-tuning methods had advanced few-shot natural language understanding tasks, self-training methods are also being explored. This work revisits the consistency regularization in self-training and presents explicit and implicit consistency regularization enhanced language model (EICO). By employing both explicit and implicit consistency regularization, EICO advances the performance of prompt-based few-shot text classification. For implicit consistency regularization, we generate pseudo-label from the weaklyaugmented view and predict pseudo-label from the strongly-augmented view. For explicit consistency regularization, we minimize the difference between the prediction of the augmentation view and the prediction of the original view. We conducted extensive experiments on six text classification datasets and found that with sixteen labeled examples, EICO achieves competitive performance compared to existing self-training few-shot learning methods.
Ask Lune
Know this paper in minutes, not an afternoon.
Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.
1 top-tier paper has cited it since. Know what they found before your reviewers do.
Lune can draw on
the full paper · the top-tier papers that cite it · related work
Free to start. No credit card required.
Lineage
- R-Drop: Regularized Dropout for Neural Networks cites Language Models are Few-Shot Learners
- Language Models are Few-Shot Learners cites Unsupervised Data Augmentation for Consistency Training
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence cites Unsupervised Data Augmentation for Consistency Training
Cited by top-tier papers1
Builds on10
- Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceKihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang et al.NeurIPS 2020 · 5,129 citations
- Unsupervised Data Augmentation for Consistency TrainingQizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong et al.NeurIPS 2020 · 2,774 citations
- SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
- R-Drop: Regularized Dropout for Neural NetworksXiaobo Liang, Lijun Wu, Juntao Li, Yue Wang et al.NeurIPS 2021 · 610 citations
Related papers
- Revisiting Self-training for Few-shot Learning of Language ModelYiming Chen, Yan Zhang, Chen Zhang, Grandee Lee et al.EMNLP 2021 · 35 citations
- Parameter-free Automatically Prompting: A Latent Pseudo Label Mapping Model for Prompt-based LearningJirui Qi, Richong Zhang, Junfan Chen, Jaein Kim et al.EMNLP Findings 2022
- Towards Unified Prompt Tuning for Few-shot Text ClassificationJianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan et al.EMNLP Findings 2022 · 37 citations
- Noisy Channel Language Model Prompting for Few-Shot Text ClassificationSewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke ZettlemoyerACL 2022 · 237 citations
- Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot LearningYu Meng, Martin Michalski, Jiaxin Huang, Yu Zhang et al.ICML 2023 · 64 citations
