FDPT: Federated Discrete Prompt Tuning for Black-Box Visual-Language Models

Vision-Language Models (VLMs) have driven significant advancements in multimodal AI. Fine-tuning these models with user data enhances adaptability but poses privacy risks. While federated learning (FL) mitigates user data privacy concerns, it fails to protect model properties. Existing methods relying on black-box VLM APIs often require access to prediction logits, making them vulnerable to inversion attacks. Additionally, optimizing tuning complexity and data transmission efficiency in federated VLM scenarios remains a challenge. To address these challenges, we propose FDPT-the first federated discrete prompt tuning method utilizing black-box VLMs. During client optimization stage, FDPT employs an agent-driven framework leveraging large language models (LLMs) with enhanced reasoning capacities to systematically optimize discrete prompt representations, and also utilizes feedback mechanisms and chain of thought to enhance prediction accuracy. Importantly, it performs optimization by relying not on the predicted logic vectors output by LLMs but on textual results, avoiding reverse attack risks. During global aggregation stage, We mimic human electoral activities by employing evolutionary computation methods underpinned by semantic similarity computation to implement enhanced zero-order optimization for acquiring representative global tokens, thereby achieving knowledge aggregation. FDPT significantly outperforms nine state-of-the-art methods in image classification and visual question-answering, reducing communication overhead while generating highly transferable optimized prompts. Additionally, it exhibits im- † Equal contribution * Corresponding author proved robustness to data heterogeneity.