  Computer Vision (CV), Natural Language Processing (NLP), and Recommender
Systems (RecSys) are three prominent AI applications that have traditionally
developed independently, resulting in disparate modeling and engineering
methodologies. This has impeded the ability for these fields to directly
benefit from each other's advancements. With the recent development of
foundation models, large language models have emerged as a potential
general-purpose interface for unifying different modalities and problem
formulations. In light of this, we propose the development of a multimodal
foundation model (MFM) considering visual, textual, and personalization
modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5),
to unify various modalities and recommendation tasks. This will enable the
processing of multiple modalities in a shared architecture for improved
recommendations. To achieve this, we introduce multimodal personalized prompts
to accommodate multiple modalities under a shared format. Additionally, we
propose a parameter-efficient training method for foundation models, which
involves freezing the P5 backbone and fine-tuning lightweight adapters,
resulting in improved recommendation performance and increased efficiency in
terms of training time and memory usage. Code and data of VIP5 are available at
https://github.com/jeykigung/VIP5.
