An Empirical Study Towards Prompt-Tuning for Graph Contrastive Pre-Training in Recommendations

Graph contrastive learning (GCL) has emerged as an effective technology for various graph learning tasks. It has been successfully applied in real-world recommender systems, where the contrastive loss and downstream recommendation objectives are combined to form the overall objective function. However, this approach deviates from the original GCL paradigm, which pre-trains graph embeddings without involving downstream training objectives. In this paper, we propose a novel framework called CPTPP, which enhances GCL-based recommender systems by leveraging prompt tuning. This framework allows us to fully exploit the advantages of the original GCL protocol. Specifically, we first summarize user profiles in graph recommender systems to automatically generate personalized user prompts. These prompts are then combined with pre-trained user embeddings for prompt tuning in downstream tasks. This helps bridge the gap between pre-training and downstream tasks. Our extensive experiments on three benchmark datasets confirm the effectiveness of CPTPP compared to state-of-the-art baselines. Additionally, a visualization experiment illustrates that user embeddings generated by CPTPP have a more uniform distribution, indicating improved modeling capability for user preferences. The implementation code is available online 2 for reproducibility. * Corresponding author. 2 https://github.com/Haoran-Young/CPTPP 37th Conference on Neural Information Processing Systems (NeurIPS 2023). not align with the purpose of GCL, which is primarily designed for pre-training graph representations without involving downstream task objectives [21, 16] . In this approach, GCL pre-trains embeddings that are then fine-tuned on specific tasks using downstream models. Incorporating both GCL and recommendation objectives into the overall training objective can disrupt the embedding pre-training process and requires careful control of the weight placed on contrastive loss. Additionally, previous studies on GCL-based recommendation methods [27, 14] have shown that the weights of contrastive loss in the overall objective are significantly smaller compared to the weight on the recommendation objective. This is done to ensure desired performance on recommendation tasks. Therefore, based on these observations, simply combining contrastive loss with downstream recommendation objectives may not be effective for recommendation tasks. The disparity between the pre-training objective and downstream tasks hinders the effective extraction of useful information from pre-trained embeddings by downstream models [12, 26] . Consequently, researchers often opt for combining GCL with recommendation objectives. However, it is important to note that GCL pre-training targets primarily assess the agreement of mutual information among graph elements, such as nodes, edges, and sub-graphs. This differs from conventional graph learning tasks like node classification and link prediction. Consequently, the pre-training targets of GCL also significantly diverge from downstream recommendation objectives that involve interaction (link) prediction between users and items. Consequently, the reduction of such dissimilarities is essential to enhance the performance of GCL-based recommendation approaches. In this paper, we present the CPTPP framework as an extension of recent advancements in prompt tuning for enhancing recommendation performance [23, 33] utilizing user embeddings pre-trained by GCL. The technique of prompt tuning has emerged as a prominent method for fine-tuning pre-trained models. By constructing appropriate prompts for downstream learning modules, this approach effectively reformulates downstream tasks, thereby reducing disparities [26, 12, 15, 18] . By incorporating prompt-tuning, we can modify existing GCL-based recommendation methods to align with the original GCL protocol involving pre-training and fine-tuning. Previous endeavors have also explored the integration of prompt learning into conventional recommendation models [26, 3] . Despite their advantages, applying the prompt mechanism directly to GCL-based recommendation methods is still difficult and not straightforward, i.e., how can we generate personalized user prompts using only the user-item interaction graph without side information (e.g., age and occupation)? To address this issue, we summarise three methods to produce different user profiles, including historical interaction records, adjacency matrix factorization, and high-order user relations, based on the user-item interaction graph for the personalized user prompt generation, which is applicable in situations devoid of side information. Comprehensive experiments conducted on three publicly available datasets illustrate the effectiveness of the proposed method with different types of prompts. In summary, the contributions of this work are three-fold: (1) We propose a reformulation of existing GCL-based recommendation methods by incorporating the promp