Lune

ACL Findings2023

When Gradient Descent Meets Derivative-Free Optimization: A Match Made in Black-Box Scenario

Chengcheng Han, Liqing Cui, Renyu Zhu, Jianing Wang, Nuo Chen, Qiushi Sun, Xiang Li, Ming Gao

Abstract

Large pre-trained language models (PLMs) have garnered significant attention for their versatility and potential for solving a wide spectrum of natural language processing (NLP) tasks. However, the cost of running these PLMs may be prohibitive. Furthermore, PLMs may not be open-sourced due to commercial considerations and potential risks of misuse, such as GPT-3. The parameters and gradients of PLMs are unavailable in this scenario. To solve the issue, black-box tuning has been proposed, which utilizes derivative-free optimization (DFO), instead of gradient descent, for training task-specific continuous prompts. However, these gradient-free methods still exhibit a significant gap compared to gradientbased methods. In this paper, we introduce gradient descent into black-box tuning scenario through knowledge distillation. Furthermore, we propose a novel method GDFO, which integrates gradient descent and derivative-free optimization to optimize task-specific continuous prompts in a harmonized manner. Experimental results show that GDFO can achieve significant performance gains over previous state-ofthe-art methods.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

1 top-tier paper has cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • Black-Box Tuning for Language-Model-as-a-Service cites Revisiting Few-sample BERT Fine-tuning
  • Black-Box Tuning for Language-Model-as-a-Service cites Language Models are Few-Shot Learners
  • Adaptive Stochastic Gradient Algorithm for Black-box Multi-Objective Learning cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers1

  1. Adaptive Stochastic Gradient Algorithm for Black-box Multi-Objective LearningFeiyang Ye, Yueming Lyu, Xuehao Wang, Yu Zhang et al.ICLR 2024 · 5 citations

Builds on16

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. TinyBERT: Distilling BERT for Natural Language UnderstandingXiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang et al.EMNLP Findings 2020 · 1,787 citations
  4. Black-Box Tuning for Language-Model-as-a-ServiceTianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang et al.ICML 2022 · 343 citations
  5. Revisiting Few-sample BERT Fine-tuningTianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Weinberger et al.ICLR 2021 · 172 citations

Related papers

  1. FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language ModelsJingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang et al.ICML 2024 · 38 citations
  2. Collaborative Discrete-Continuous Black-Box Prompt Learning for Language ModelsHualin Zhang, Haozhen Zhang, Zhekai Liu, Bin Gu et al.ICLR 2025
  3. FedOne: Query-Efficient Federated Learning for Black-box Discrete Prompt LearningGanyu Wang, Jinjie Fang, Maxwell J. Yin, Bin Gu et al.ICML 2025
  4. CBP-Tuning: Efficient Local Customization for Black-box Large Language ModelsJiaxuan Zhao, Naibin Gu, Yuchen Feng, Xiyu Liu et al.EMNLP 2025
  5. Leveraging First and Zeroth-Order Gradient to Address Imbalanced Black-Box Prompt Tuning via Minimax OptimizationHaozhen Zhang, Zhaogeng Liu, Bin Gu, Yi ChangAAAI 2025 · 1 citation
Dusk over the sea between two cliffs drawn in fine vertical lines