  Multimodal learning aims to discover the relationship between multiple
modalities. It has become an important research topic due to extensive
multimodal applications such as cross-modal retrieval. This paper attempts to
address the modality heterogeneity problem based on Gaussian process latent
variable models (GPLVMs) to represent multimodal data in a common space.
Previous multimodal GPLVM extensions generally adopt individual learning
schemes on latent representations and kernel hyperparameters, which ignore
their intrinsic relationship. To exploit strong complementarity among different
modalities and GPLVM components, we develop a novel learning scheme called
Harmonization, where latent model parameters are jointly learned from each
other. Beyond the correlation fitting or intra-modal structure preservation
paradigms widely used in existing studies, the harmonization is derived in a
model-driven manner to encourage the agreement between modality-specific GP
kernels and the similarity of latent representations. We present a range of
multimodal learning models by incorporating the harmonization mechanism into
several representative GPLVM-based approaches. Experimental results on four
benchmark datasets show that the proposed models outperform the strong
baselines for cross-modal retrieval tasks, and that the harmonized multimodal
learning method is superior in discovering semantically consistent latent
representation.
