  We are perceiving and communicating with the world in a multisensory manner,
where different information sources are sophisticatedly processed and
interpreted by separate parts of the human brain to constitute a complex, yet
harmonious and unified sensing system. To endow the machines with true
intelligence, multimodal machine learning that incorporates data from various
sources has become an increasingly popular research area with emerging
technical advances in recent years. In this paper, we present a survey on
multimodal machine learning from a novel perspective considering not only the
purely technical aspects but also the intrinsic nature of different data
modalities. We analyze the commonness and uniqueness of each data format mainly
ranging from vision, audio, text, and motions, and then present the
methodological advancements categorized by the combination of data modalities,
such as Vision+Text, with slightly inclined emphasis on the visual data. We
investigate the existing literature on multimodal learning from both the
representation learning and downstream application levels, and provide an
additional comparison in the light of their technical connections with the data
nature, e.g., the semantic consistency between image objects and textual
descriptions, and the rhythm correspondence between video dance moves and
musical beats. We hope that the exploitation of the alignment as well as the
existing gap between the intrinsic nature of data modality and the technical
designs, will benefit future research studies to better address a specific
challenge related to the concrete multimodal task, prompting a unified
multimodal machine learning framework closer to a real human intelligence
system.
