Lune

ACL Findings2023

Rethinking Semi-supervised Learning with Language Models

Zhengxiang Shi, Francesco Tonolini, Nikolaos Aletras, Emine Yilmaz, Gabriella Kazai, Yunlong Jiao

Abstract

Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks. Currently, there are two popular approaches to make use of unlabelled data: Self-training (ST) and Task-adaptive pre-training (TAPT). ST uses a teacher model to assign pseudo-labels to the unlabelled data, while TAPT continues pre-training on the unlabelled data before finetuning. To the best of our knowledge, the effectiveness of TAPT in SSL tasks has not been systematically studied, and no previous work has directly compared TAPT and ST in terms of their ability to utilize the pool of unlabelled data. In this paper, we provide an extensive empirical study comparing five state-of-the-art ST approaches and TAPT across various NLP tasks and data sizes, including in-and out-of-domain settings. Surprisingly, we find that TAPT is a strong and more robust SSL learner, even when using just a few hundred unlabelled samples or in the presence of domain shifts, compared to more sophisticated ST approaches, and tends to bring greater improvements in SSL than in fully-supervised settings. Our further analysis demonstrates the risks of using ST approaches when the size of labelled or unlabelled data is small or when domain shifts exist. We offer a fresh perspective for future SSL research, suggesting the use of unsupervised pre-training objectives over dependency on pseudo labels. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling cites ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring
  • FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling cites Unsupervised Data Augmentation for Consistency Training
  • FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling cites FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
  • Rethinking Pre-training and Self-training cites Unsupervised Data Augmentation for Consistency Training
  • Rethinking Pre-training and Self-training cites FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
  • Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner cites FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling
  • Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner cites ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring
  • Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner cites Unsupervised Data Augmentation for Consistency Training
  • Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner cites DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
  • DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning cites Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
  • Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner cites FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
  • ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring cites Unsupervised Data Augmentation for Consistency Training
  • Instruction Tuning With Loss Over Instructions cites Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
  • Instruction Tuning With Loss Over Instructions cites DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuning
  • FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence cites ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation Anchoring
  • FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence cites Unsupervised Data Augmentation for Consistency Training
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Instruction Tuning With Loss Over InstructionsZhengxiang Shi, Adam X. Yang, Bin Wu, Laurence Aitchison et al.NeurIPS 2024 · 55 citations
  2. DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningZhengxiang Shi, Aldo LipaniICLR 2024 · 46 citations
  3. Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful LearnerZhengxiang Shi, Aldo LipaniNeurIPS 2023 · 36 citations
  4. BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor AttacksIvan Sabolic, Marin Oršić, Josip Šarić, Sven LoncaricICML 2026
  5. Towards Anomaly-Aware Pre-Training and Fine-Tuning for Graph Anomaly DetectionYunhui Liu, Jiashun Cheng, Yiqing Lin, Qizhuo Xie et al.ICLR 2026

Builds on21

  1. FixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceKihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang et al.NeurIPS 2020 · 5,129 citations
  2. Unsupervised Data Augmentation for Consistency TrainingQizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong et al.NeurIPS 2020 · 2,774 citations
  3. FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo LabelingBowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu et al.NeurIPS 2021 · 1,389 citations
  4. Rethinking Pre-training and Self-trainingBarret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui et al.NeurIPS 2020 · 755 citations
  5. ReMixMatch: Semi-Supervised Learning with Distribution Matching and Augmentation AnchoringDavid Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin et al.ICLR 2020 · 469 citations

Related papers

  1. Task-adaptive Pre-training and Self-training are Complementary for Natural Language UnderstandingShiyang Li, Semih Yavuz, Wenhu Chen, Xifeng YanEMNLP Findings 2021 · 9 citations
  2. Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language ModelsChristopher Schröder, Gerhard HeyerEMNLP 2024 · 2 citations
  3. Self-Tuning for Data-Efficient Deep LearningXimei Wang, Jinghan Gao, Mingsheng Long, Jianmin WangICML 2021 · 79 citations
  4. Leveraging Training Dynamics and Self-Training for Text ClassificationTiberiu Sosea, Cornelia CarageaEMNLP Findings 2022 · 11 citations
  5. STraTA: Self-Training with Task Augmentation for Better Few-shot LearningTu Vu, Minh-Thang Luong, Quoc V. Le, Grady Simon et al.EMNLP 2021 · 25 citations
Dusk over the sea between two cliffs drawn in fine vertical lines