Lune

EMNLP Findings2023

Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification

Yookyung Kho, Jaehee Kim, Pilsung Kang

Abstract

Recently, prompt-based fine-tuning has garnered considerable interest as a core technique for few-shot text classification task. This approach reformulates the fine-tuning objective to align with the Masked Language Modeling (MLM) objective. Leveraging unlabeled data, prompt-based self-training has shown greater effectiveness in binary and three-class classification. However, prompt-based self-training for multi-class classification has not been adequately investigated, despite its significant applicability to real-world scenarios. Moreover, extending current methods to multi-class classification suffers from the verbalizer that extracts the predicted value of manually pre-defined single label word for each class from MLM predictions. Consequently, we introduce a novel, efficient verbalizer structure, named Mappingfree Automatic Verbalizer (MAV). Comprising two fully connected layers, MAV serves as a trainable verbalizer that automatically extracts the requisite word features for classification by capitalizing on all available information from MLM predictions. Experimental results on five multi-class classification datasets indicate MAV's superior self-training efficacy. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling cites Unsupervised Data Augmentation for Consistency Training
  • FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling cites FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
  • FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence cites Unsupervised Data Augmentation for Consistency Training
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on19

  1. FixMatch: Simplifying Semi-Supervised Learning with Consistency and ConfidenceKihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang et al.NeurIPS 2020 · 5,129 citations
  2. Unsupervised Data Augmentation for Consistency TrainingQizhe Xie, Zihang Dai, Eduard H. Hovy, Thang Luong et al.NeurIPS 2020 · 2,774 citations
  3. SimCSE: Simple Contrastive Learning of Sentence EmbeddingsTianyu Gao, Xingcheng Yao, Danqi ChenEMNLP 2021 · 2,496 citations
  4. FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo LabelingBowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu et al.NeurIPS 2021 · 1,389 citations
  5. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations

Related papers

  1. SciPrompt: Knowledge-augmented Prompting for Fine-grained Categorization of Scientific TopicsZhiwen You, Kanyao Han, Haotian Zhu, Bertram Ludäscher et al.EMNLP 2024
  2. Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text ClassificationShengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu et al.ACL 2022
  3. Towards Unified Prompt Tuning for Few-shot Text ClassificationJianing Wang, Chengyu Wang, Fuli Luo, Chuanqi Tan et al.EMNLP Findings 2022 · 37 citations
  4. MetricPrompt: Prompting Model as a Relevance Metric for Few-shot Text ClassificationHongyuan Dong, Weinan Zhang, Wanxiang CheKDD 2023 · 2 citations
  5. Hierarchical Verbalizer for Few-Shot Hierarchical Text ClassificationKe Ji, Yixin Lian, Jingsheng Gao, Baoyuan WangACL 2023 · 17 citations
Dusk over the sea between two cliffs drawn in fine vertical lines