Lune

EMNLP Findings2024

StablePT : Towards Stable Prompting for Few-shot Learning via Input Separation

Xiaoming Liu, Chen Liu, Zhaohan Zhang, Chengzhengxu Li, Longtian Wang, Yu Lan, Chao Shen

Abstract

Large language models have shown their ability to become effective few-shot learners with prompting, revolutionizing the paradigm of learning with data scarcity. However, this approach largely depends on the quality of prompt initialization, and always exhibits large variability among different runs. Such property makes prompt tuning highly unreliable and vulnerable to poorly constructed prompts, which limits its extension to more real-world applications. To tackle this issue, we propose to treat the hard prompt and soft prompt as separate inputs to mitigate noise brought by the prompt initialization. Furthermore, we optimize soft prompts with contrastive learning for utilizing class-aware information in the training process to maintain model performance. Experimental results demonstrate that StablePT outperforms state-of-the-art methods by 6.97% in accuracy and reduces the standard deviation by 1.92 on average. Furthermore, extensive experiments underscore its robustness and stability across 8 datasets covering various tasks. 1

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
  • Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models cites Language Models are Few-Shot Learners
  • Training language models to follow instructions with human feedback cites Language Models are Few-Shot Learners
  • Calibrate Before Use: Improving Few-shot Performance of Language Models cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language ModelsChengzhengxu Li, Xiaoming Liu, Zhaohan Zhang, Yichen Wang et al.NeurIPS 2024 · 8 citations

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Training language models to follow instructions with human feedbackLong Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida et al.NeurIPS 2022 · 24,707 citations
  3. Calibrate Before Use: Improving Few-shot Performance of Language ModelsZihao Zhao, Eric Wallace, Shi Feng, Dan Klein et al.ICML 2021 · 1,843 citations
  4. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  5. Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningBeliz Gunel, Jingfei Du, Alexis Conneau, Veselin StoyanovICLR 2021 · 595 citations

Related papers

  1. Hard Sample Aware Prompt-TuningYuanjian Xu, Qi An, Jiahuan Zhang, Peng Li et al.ACL 2023 · 4 citations
  2. PPT: Pre-trained Prompt Tuning for Few-shot LearningYuxian Gu, Xu Han, Zhiyuan Liu, Minlie HuangACL 2022
  3. Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPOXin Yang, Letian Li, Abudukelimu Wuerkaixi, Xuxin Cheng et al.ICLR 2026 · 6 citations
  4. The Power of Scale for Parameter-Efficient Prompt TuningBrian Lester, Rami Al-Rfou, Noah ConstantEMNLP 2021 · 94 citations
  5. CPL: Counterfactual Prompt Learning for Vision and Language ModelsXuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu et al.EMNLP 2022 · 13 citations
Dusk over the sea between two cliffs drawn in fine vertical lines