FT-PromptFL: A Feature Transmission-based Framework for Communication-Efficient Prompt Federated Learning

PromptFL combines prompt tuning and federated learning to enable collaborative model training. It employs the entire Contrastive Language-Image Pre-training (CLIP) model on the client side, and submits the prompt learner for global aggregation. However, the large scale of the prompt network results in significant communication overhead per round, a challenge that remains unaddressed. To tackle this issue, this study presents a Feature Transmission-based PromptFL (FT-PromptFL), which distributes the CLIP encoder across clients and the server. Furthermore, we propose a Federated Feature Compaction (FFC) framework to minimize communication costs by constructing compact feature subsets through federated clustering. Theoretical analysis is provided to demonstrate the effectiveness of FFC. To mitigate performance degradation caused by compressed transmission, Semantic Compensation (SC) mechanism is designed through the multi-modal fusion of partial images and intact text. In addition, a dynamic weighting strategy is introduced to constrain the generalization error arising from asymmetric modality fusion. Thus, the model performance can be guaranteed while maintaining low communication cost. Experimental results show that the communication overhead of FT-PromptFL decreases 90% per-round compared to PromptFL. It also achieves superior inference performance, such as state-of-the-art (SOTA) 96.85% accuracy on the Flowers102 dataset.