Lune

EMNLP Findings2022

Clip-Tuning: Towards Derivative-free Prompt Learning with a Mixture of Rewards

Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

Abstract

Derivative-free prompt learning has emerged as a lightweight alternative to prompt tuning, which only requires model inference to optimize the prompts. However, existing work did not take full advantage of the over-parameterized characteristics of large pretrained language models (PLMs). In this paper, we propose Clip-Tuning, a simple yet effective method that adopts diverse frozen "thinned" networks of PLMs to obtain a mixture of rewards and thus advance the derivative-free prompt learning. The thinned networks consist of all the hidden units that survive a stationary dropout strategy, whose inference predictions reflect an ensemble of partial views over prompted training samples. Our method outperforms previous gradient-free prompt learning methods and achieves parity with gradientbased counterparts on seven language understanding benchmarks under few-shot settings.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

9 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with Transformers cites Language Models are Few-Shot Learners
  • Fine-Tuning Language Models with Just Forward Passes cites LoRA: Low-Rank Adaptation of Large Language Models
  • Fine-Tuning Language Models with Just Forward Passes cites Language Models are Few-Shot Learners
  • Fine-Tuning Language Models with Just Forward Passes cites PromptBoosting: Black-Box Text Classification with Ten Forward Passes
  • Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark cites Fine-Tuning Language Models with Just Forward Passes
  • Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark cites The Lottery Ticket Hypothesis for Pre-trained BERT Networks
  • The Lottery Ticket Hypothesis for Pre-trained BERT Networks cites Language Models are Few-Shot Learners
  • AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts cites Language Models are Few-Shot Learners
  • R-Drop: Regularized Dropout for Neural Networks cites Language Models are Few-Shot Learners
  • Language Models as Black-Box Optimizers for Vision-Language Models cites AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
  • Language Models as Black-Box Optimizers for Vision-Language Models cites Language Models are Few-Shot Learners
  • PromptBoosting: Black-Box Text Classification with Ten Forward Passes cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers9

  1. Fine-Tuning Language Models with Just Forward PassesSadhika Malladi, Tianyu Gao, Eshaan Nichani, Alex Damian et al.NeurIPS 2023 · 495 citations
  2. Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A BenchmarkYihua Zhang, Pingzhi Li, Junyuan Hong, Jiaxiang Li et al.ICML 2024 · 134 citations
  3. PromptBoosting: Black-Box Text Classification with Ten Forward PassesBairu Hou, Joe O'Connor, Jacob Andreas, Shiyu Chang et al.ICML 2023 · 56 citations
  4. Use Your INSTINCT: INSTruction optimization for LLMs usIng Neural bandits Coupled with TransformersXiaoqiang Lin, Zhaoxuan Wu, Zhongxiang Dai, Wenyang Hu et al.ICML 2024 · 26 citations
  5. Language Models as Black-Box Optimizers for Vision-Language ModelsShihong Liu, Samuel Yu, Zhiqiu Lin, Deepak Pathak et al.CVPR 2024 · 18 citations

Builds on11

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated PromptsTaylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace et al.EMNLP 2020 · 1,162 citations
  4. R-Drop: Regularized Dropout for Neural NetworksXiaobo Liang, Lijun Wu, Juntao Li, Yue Wang et al.NeurIPS 2021 · 610 citations
  5. The Lottery Ticket Hypothesis for Pre-trained BERT NetworksTianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu et al.NeurIPS 2020 · 428 citations

Related papers

  1. Few-Shot Composition Learning for Image Retrieval with Prompt TuningJunda Wu, Rui Wang, Handong Zhao, Ruiyi Zhang et al.AAAI 2023 · 16 citations
  2. Aggregate-and-Adapt Natural Language Prompts for Downstream Generalization of CLIPChen Huang, Skyler Seto, Samira Abnar, David Grangier et al.NeurIPS 2024 · 8 citations
  3. PPT: Pre-trained Prompt Tuning for Few-shot LearningYuxian Gu, Xu Han, Zhiyuan Liu, Minlie HuangACL 2022
  4. Black-Box Test-Time Prompt Tuning for Vision-Language ModelsFan'an Meng, Chaoran Cui, Hongjun Dai, Shuai GongAAAI 2025 · 6 citations
  5. Pre-trained Language Models Can be Fully Zero-Shot LearnersXuandong Zhao, Siqi Ouyang, Zhiguo Yu, Ming Wu et al.ACL 2023 · 22 citations
Dusk over the sea between two cliffs drawn in fine vertical lines