Selective Prefix Tuning for Pre-trained Language Models

The prevalent approach for optimizing pretrained language models in downstream tasks is fine-tuning. However, it is both timeconsuming and memory-inefficient. In response, a more efficient method called Prefix Tuning, which inserts learnable vectors into each Transformer layers, has been proposed and proven effective. Recent investigations reveal that prefix tokens carry context-specific information, prompting the hypothesis that enhancing their specialization can improve model performance. To address this, we propose Selective Prefix Tuning (SPT), integrating a selective mechanism inspired by selective selfattention. Additionally, we introduce Selective Loss (SL) to encourage diversity in prefix tokens. Extensive experiments validate the effectiveness of SPT in sentence and token classification tasks. We contribute insight into understanding the role of prefix in model adaptation.