  Despite the recognized potential of multimodal data to improve model
accuracy, many large-scale industrial recommendation systems, including Taobao
display advertising system, predominantly depend on sparse ID features in their
models. In this work, we explore approaches to leverage multimodal data to
enhance the recommendation accuracy. We start from identifying the key
challenges in adopting multimodal data in a manner that is both effective and
cost-efficient for industrial systems. To address these challenges, we
introduce a two-phase framework, including: 1) the pre-training of multimodal
representations to capture semantic similarity, and 2) the integration of these
representations with existing ID-based models. Furthermore, we detail the
architecture of our production system, which is designed to facilitate the
deployment of multimodal representations. Since the integration of multimodal
representations in mid-2023, we have observed significant performance
improvements in Taobao display advertising system. We believe that the insights
we have gathered will serve as a valuable resource for practitioners seeking to
leverage multimodal data in their systems.
