  Recently there has been a significant surge in multimodal learning in terms
of both image-to-text and text-to-image generation. However, the success is
typically limited to English, leaving other languages largely behind. Building
a competitive counterpart in other languages is highly challenging due to the
low-resource nature of non-English multimodal data (i.e., lack of large-scale,
high-quality image-text data). In this work, we propose MPM, an effective
training paradigm for training large multimodal models in non-English
languages. MPM demonstrates that Multilingual language models can Pivot
zero-shot Multimodal learning across languages. Specifically, based on a strong
multilingual large language model, multimodal models pretrained on English-only
image-text data can well generalize to other languages in a (quasi)-zero-shot
manner, even surpassing models trained on image-text data in native languages.
Taking Chinese as a practice of MPM, we build large multimodal models VisCPM in
image-to-text and text-to-image generation, which achieve state-of-the-art
(open-source) performance in Chinese. To facilitate future research, we
open-source codes and model weights at https://github.com/OpenBMB/VisCPM.git.
