Lune

EMNLP Findings2023

Tunable Soft Prompts are Messengers in Federated Learning

Chenhe Dong, Yuexiang Xie, Bolin Ding, Ying Shen, Yaliang Li

Abstract

Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources, alleviating privacy concerns that arise from directly sharing local data. However, the lack of model privacy protection in FL becomes an unneglectable challenge, especially when people want to federally finetune models based on a proprietary large language model. In this study, we propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts. These soft prompts, updated and transmitted between the server and clients, assume the role of the global model parameters and serve as messengers to deliver useful knowledge from the local data and global model. As the global model itself is not required to be shared and the local training is conducted based on an auxiliary model with fewer parameters than the global model, the proposed approach provides protection for the global model while reducing communication and computation costs in FL. Extensive experiments show the effectiveness of the proposed approach compared to several baselines. We have released the source code at https: //github.com/alibaba/FederatedScope/ tree/fedsp/federatedscope/nlp/fedsp.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

Questions to start from

Lune can draw on

the full paper · related work

Get the answer

Free to start. No credit card required.

Lineage

  • LoRA: Low-Rank Adaptation of Large Language Models cites Language Models are Few-Shot Learners
  • GLM-130B: An Open Bilingual Pre-trained Model cites Language Models are Few-Shot Learners
  • Towards a Unified View of Parameter-Efficient Transfer Learning cites Language Models are Few-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Builds on11

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. LoRA: Low-Rank Adaptation of Large Language ModelsEdward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu et al.ICLR 2022 · 18,833 citations
  3. ALBERT: A Lite BERT for Self-supervised Learning of Language RepresentationsZhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel et al.ICLR 2020 · 7,418 citations
  4. Towards a Unified View of Parameter-Efficient Transfer LearningJunxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick et al.ICLR 2022 · 1,182 citations
  5. GLM-130B: An Open Bilingual Pre-trained ModelAohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang et al.ICLR 2023 · 295 citations

Related papers

  1. FedBPT: Efficient Federated Black-box Prompt Tuning for Large Language ModelsJingwei Sun, Ziyue Xu, Hongxu Yin, Dong Yang et al.ICML 2024 · 38 citations
  2. Promoting Data and Model Privacy in Federated Learning through Quantized LoRAJianhao Zhu, Changze Lv, Xiaohua Wang, Muling Wu et al.EMNLP Findings 2024 · 4 citations
  3. Enhancing Model Privacy in Federated Learning with Random Masking and QuantizationZhibo Xu, Jianhao Zhu, Jingwen Xu, Changze Lv et al.EMNLP Findings 2025 · 1 citation
  4. Can Federated Learning Safeguard Private Data in LLM Training? Vulnerabilities, Attacks, and Defense EvaluationWenkai Guo, Xuefeng Liu, Haolin Wang, Jianwei Niu et al.EMNLP Findings 2025
  5. FedPETuning: When Federated Learning Meets the Parameter-Efficient Tuning Methods of Pre-trained Language ModelsZhuo Zhang, Yuanhang Yang, Yong Dai, Qifan Wang et al.ACL Findings 2023 · 98 citations
Dusk over the sea between two cliffs drawn in fine vertical lines