Lune

EMNLP Findings2023

Black-Box Tuning of Vision-Language Models with Effective Gradient Approximation

Zixian Guo, Yuxiang Wei, Ming Liu, Zhilong Ji, Jinfeng Bai, Yiwen Guo, Wangmeng Zuo

Abstract

Parameter-efficient fine-tuning (PEFT) methods have provided an effective way for adapting large vision-language models to specific tasks or scenarios. Typically, they learn a very small scale of parameters for pre-trained models in a white-box formulation, which assumes model architectures to be known and parameters to be accessible. However, large models are often not open-source due to considerations of preventing abuse or commercial factors, hence posing a barrier to the deployment of white-box PEFT methods. To alleviate the dependence on model accessibility, we introduce collaborative black-box tuning (CBBT) for both textual prompt optimization and output feature adaptation for black-box models. Specifically, considering that the backpropagation gradients are blocked, we approximate the gradients of textual prompts by analyzing the predictions with perturbed prompts. Secondly, a lightweight adapter is deployed over the output feature of the inaccessible model, further facilitating the model adaptation process. Empowered with these designs, our CBBT is extensively evaluated on eleven downstream benchmarks and achieves remarkable improvements compared to existing black-box VL adaptation methods. Our code will be made publicly available.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

6 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • Training Spatial-Frequency Visual Prompts and Probabilistic Clusters for Accurate Black-Box Transfer Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
  • Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision cites Learning Transferable Visual Models From Natural Language Supervision
  • Black-Box Forgetting cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Black-Box Forgetting cites Learning Transferable Visual Models From Natural Language Supervision
  • Black-Box Test-Time Prompt Tuning for Vision-Language Models cites Learning Transferable Visual Models From Natural Language Supervision
  • Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts cites Learning Transferable Visual Models From Natural Language Supervision
  • FILIP: Fine-grained Interactive Language-Image Pre-Training cites Large Batch Optimization for Deep Learning: Training BERT in 76 minutes
  • FILIP: Fine-grained Interactive Language-Image Pre-Training cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • FILIP: Fine-grained Interactive Language-Image Pre-Training cites Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites FILIP: Fine-grained Interactive Language-Image Pre-Training
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model Adaptation cites Learning Transferable Visual Models From Natural Language Supervision
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers6

  1. Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic PromptsZhi-Yi Chin, Chieh-Ming Jiang, Ching-Chun Huang, Pin-Yu Chen et al.ICML 2024 · 155 citations
  2. Black-Box ForgettingYusuke Kuwana, Yuta Goto, Takashi Shibata, Go IrieNeurIPS 2024 · 6 citations
  3. Black-Box Test-Time Prompt Tuning for Vision-Language ModelsFan'an Meng, Chaoran Cui, Hongjun Dai, Shuai GongAAAI 2025 · 6 citations
  4. Prime Once, then Reprogram Locally: An Efficient Alternative to Black-Box Service Model AdaptationYunbei Zhang, Chengyi Cai, Feng Liu, Jihun HammCVPR 2026 · 5 citations
  5. Training Spatial-Frequency Visual Prompts and Probabilistic Clusters for Accurate Black-Box Transfer LearningWonwoo Cho, Kangyeol Kim, Saemee Choi, Jaegul ChooACM MM 2024

Builds on16

  1. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  2. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  3. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text SupervisionChao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen et al.ICML 2021 · 5,401 citations
  4. Large Batch Optimization for Deep Learning: Training BERT in 76 minutesYang You, Jing Li, Sashank J. Reddi, Jonathan Hseu et al.ICLR 2020 · 1,170 citations
  5. FILIP: Fine-grained Interactive Language-Image Pre-TrainingLewei Yao, Runhui Huang, Lu Hou, Guansong Lu et al.ICLR 2022 · 827 citations

Related papers

  1. Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language ModelsZhengbo Wang, Jian Liang, Ran He, Zilei Wang et al.ICML 2024
  2. FATE: Feature-Adapted Parameter Tuning for Vision-Language ModelsZhengqin Xu, Zelin Peng, Xiaokang Yang, Wei ShenAAAI 2025 · 3 citations
  3. CVPT: Cross Visual Prompt TuningLingyun Huang, Jianxu Mao, Junfei Yi, Ziming Tao et al.ICCV 2025 · 6 citations
  4. CombLM: Adapting Black-Box Language Models through Small Fine-Tuned ModelsAitor Ormazabal, Mikel Artetxe, Eneko AgirreEMNLP 2023 · 3 citations
  5. FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language ModelsJingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang et al.ICML 2024 · 38 citations
Dusk over the sea between two cliffs drawn in fine vertical lines