Lune

EMNLP Findings2023

VIP5: Towards Multimodal Foundation Models for Recommendation

Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, Yongfeng Zhang

Abstract

Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys) are three prominent AI applications that have traditionally developed independently, resulting in disparate modeling and engineering methodologies. This has impeded the ability for these fields to directly benefit from each other's advancements. With the recent development of foundation models, large language models have emerged as a potential generalpurpose interface for unifying different modalities and problem formulations. In light of this, we propose the development of a multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks. This will enable the processing of multiple modalities in a shared architecture for improved recommendations. To achieve this, we introduce multimodal personalized prompts to accommodate multiple modalities under a shared format. Additionally, we propose a parameter-efficient training method for foundation models, which involves freezing the P5 backbone and fine-tuning lightweight adapters, resulting in improved recommendation performance and increased efficiency in terms of training time and memory usage. Code and data of VIP5 are available at https:// github.com/jeykigung/VIP5.

Ask Lune

Know this paper in minutes, not an afternoon.

Lune can read every page of it and the top-tier papers that cite it, and gives you its claims, its evidence and its limits, each one cited to the paper it comes from. No skimming, no AI slop.

17 top-tier papers have cited it since. Know what they found before your reviewers do.

Questions to start from

Lune can draw on

the full paper · the top-tier papers that cite it · related work

Get the answer

Free to start. No credit card required.

Lineage

  • On Softmax Direct Preference Optimization for Recommendation cites Language Models are Few-Shot Learners
  • On Softmax Direct Preference Optimization for Recommendation cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond cites Learning Transferable Visual Models From Natural Language Supervision
  • Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond cites Flamingo: a Visual Language Model for Few-Shot Learning
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Language Models are Few-Shot Learners
  • Multitask Prompted Training Enables Zero-Shot Task Generalization cites Finetuned Language Models are Zero-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT cites Language Models are Few-Shot Learners
  • IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFT cites Learning Transferable Visual Models From Natural Language Supervision
  • Learning Transferable Visual Models From Natural Language Supervision cites Language Models are Few-Shot Learners
  • Finetuned Language Models are Zero-Shot Learners cites Language Models are Few-Shot Learners
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Multitask Prompted Training Enables Zero-Shot Task Generalization
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Language Models are Few-Shot Learners
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Learning Transferable Visual Models From Natural Language Supervision
  • Flamingo: a Visual Language Model for Few-Shot Learning cites Finetuned Language Models are Zero-Shot Learners
Builds onCited byLeft to right by year; the more cited, the higher.

Cited by top-tier papers17

  1. On Softmax Direct Preference Optimization for RecommendationYuxin Chen, Junfei Tan, An Zhang, Zhengyi Yang et al.NeurIPS 2024 · 126 citations
  2. Wukong: Towards a Scaling Law for Large-Scale RecommendationBuyun Zhang, Liang Luo, Yuxin Chen, Jade Nie et al.ICML 2024 · 108 citations
  3. Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and BeyondTianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng et al.ICLR 2024 · 46 citations
  4. IDGenRec: LLM-RecSys Alignment with Textual ID LearningJuntao Tan, Shuyuan Xu, Wenyue Hua, Yingqiang Ge et al.SIGIR 2024 · 45 citations
  5. IISAN: Efficiently Adapting Multimodal Representation for Sequential Recommendation with Decoupled PEFTJunchen Fu, Xuri Ge, Xin Xin, Alexandros Karatzoglou et al.SIGIR 2024 · 40 citations

Builds on20

  1. Language Models are Few-Shot LearnersTom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah et al.NeurIPS 2020 · 64,255 citations
  2. Learning Transferable Visual Models From Natural Language SupervisionAlec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh et al.ICML 2021 · 47,906 citations
  3. Flamingo: a Visual Language Model for Few-Shot LearningJean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech et al.NeurIPS 2022 · 6,707 citations
  4. Finetuned Language Models are Zero-Shot LearnersJason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu et al.ICLR 2022 · 4,966 citations
  5. Multitask Prompted Training Enables Zero-Shot Task GeneralizationVictor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach et al.ICLR 2022 · 1,976 citations

Related papers

  1. Scaling Collaborative Filtering with Multimodal Contrastive Fine-tuningLiwei Jin, Dan Luo, Lixin Zou, Chenliang Li et al.WWW 2026
  2. Empowering Visible-Infrared Person Re-Identification with Large Foundation ModelsZhangyi Hu, Bin Yang, Mang YeNeurIPS 2024 · 45 citations
  3. mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionsChenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang et al.EMNLP 2022 · 159 citations
  4. ViM: Vision Middleware for Unified Downstream TransferringYutong Feng, Biao Gong, Jianwen Jiang, Yiliang Lv et al.ICCV 2023 · 2 citations
  5. Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual TokenizationYang Jin, Kun Xu, Liwei Chen, Chao Liao et al.ICLR 2024 · 87 citations
Dusk over the sea between two cliffs drawn in fine vertical lines