ROMA: Recommendation-Oriented Language Model Adaptation Using Multi-Modal Multi-Domain Item Sequences

Sequential recommendation (SR) aims to capture dynamic user preferences from users' historical behaviors. Recently, benefiting from astonishing understanding ability of pre-trained language models (PLMs), text-enhanced sequential recommender becomes a promising direction, which employs PLMs to extract semantic information for user/item representation. Although promising in improving performance and transferability, few existing text-enhanced SR studies have analyzed the differences between PLMs and recommenders, restricting the ability of PLMs for recommendation. In this paper, we make an in-depth comparison and conclude their discrepancies in representation and knowledge level, respectively, caused by different multi-modal content and task-oriented capabilities. Based on this, we propose a Recommendation-Oriented Language Model Adaptation framework (named ROMA) using multi-modal multi-domain item sequences. To empower PLMs with a rational understanding of user/item modeling and the recommendation task, ROMA partitions a PLM into bottom and top layers, respectively, allowing representation-level and task-level adaptation with elaborately designed architectures, transferring strategy and learning framework. Our experimental results on public benchmarks demonstrate the effectiveness and transferability of our framework. Additionally, we showcase the application value of ROMA on the recommender system of Huawei's AppGallery through online A/B testing, which shows significant improvements in online metrics.