Revisit Visual Prompt Tuning: The Expressiveness of Prompt Experts

Visual Prompt Tuning (VPT) has proven effective for parameter-efficient adaptation of pre-trained vision models to downstream tasks by inserting task-specific learnable prompt tokens. Despite its empirical success, a comprehensive theoretical understanding of VPT remains an active area of research. Building on the recently established connection between Mixture of Experts (MoE) and prompt-based methods, wherein each attention head can be conceptualized as a composition of multiple MoE models, we reinterpret VPT as the introduction of new prompt experts into these MoE structures. We identify a key limitation in existing VPT frameworks: the restricted functional expressiveness of prompt experts, which remain static and thus limited in their adaptability. To address this, we propose Visual Adaptive Prompt Tuning (VAPT), a novel method that endows prompt experts with enhanced expressiveness while preserving parameter efficiency. Empirical evaluations on VTAB-1K and FGVC demonstrate that VAPT achieves substantial performance improvements, surpassing fully fine-tuned baselines by 7.34% and 1.04%, respectively. Moreover, VAPT consistently outperforms VPT while requiring fewer additional parameters. Furthermore, our theoretical analysis indicates that VAPT achieves optimal sample efficiency. Collectively, these results underscore the theoretical grounding and empirical advantages of our approach. Our code is publicly available at https://github.com/Minhchuyentoancbn/VAPT . Recently, Le et al. (2024) established a formal connection between attention mechanisms (Vaswani, 2017) , prompt-based methods, and Mixture of Experts (MoE) models (Jacobs et al., 1991; Shazeer et al., 2017) , yielding new insights into the design and optimization of prompting strategies. Their analysis demonstrates that each attention head in a transformer can be equivalently interpreted as a composition of multiple MoE models stacked together. Within this framework, VPT corresponds to fine-tuning these implicit, pre-trained MoE structures by introducing new, learnable prompt experts. These prompt experts collaborate with the pre-trained experts to facilitate effective task adaptation.