  In this paper, we address multi-modal pretraining of product data in the
field of E-commerce. Current multi-modal pretraining methods proposed for image
and text modalities lack robustness in the face of modality-missing and
modality-noise, which are two pervasive problems of multi-modal product data in
real E-commerce scenarios. To this end, we propose a novel method, K3M, which
introduces knowledge modality in multi-modal pretraining to correct the noise
and supplement the missing of image and text modalities. The modal-encoding
layer extracts the features of each modality. The modal-interaction layer is
capable of effectively modeling the interaction of multiple modalities, where
an initial-interactive feature fusion model is designed to maintain the
independence of image modality and text modality, and a structure aggregation
module is designed to fuse the information of image, text, and knowledge
modalities. We pretrain K3M with three pretraining tasks, including masked
object modeling (MOM), masked language modeling (MLM), and link prediction
modeling (LPM). Experimental results on a real-world E-commerce dataset and a
series of product-based downstream tasks demonstrate that K3M achieves
significant improvements in performances than the baseline and state-of-the-art
methods when modality-noise or modality-missing exists.
