  In emotion recognition, it is difficult to recognize human's emotional states
using just a single modality. Besides, the annotation of physiological
emotional data is particularly expensive. These two aspects make the building
of effective emotion recognition model challenging. In this paper, we first
build a multi-view deep generative model to simulate the generative process of
multi-modality emotional data. By imposing a mixture of Gaussians assumption on
the posterior approximation of the latent variables, our model can learn the
shared deep representation from multiple modalities. To solve the
labeled-data-scarcity problem, we further extend our multi-view model to
semi-supervised learning scenario by casting the semi-supervised classification
problem as a specialized missing data imputation task. Our semi-supervised
multi-view deep generative framework can leverage both labeled and unlabeled
data from multiple modalities, where the weight factor for each modality can be
learned automatically. Compared with previous emotion recognition methods, our
method is more robust and flexible. The experiments conducted on two real
multi-modal emotion datasets have demonstrated the superiority of our framework
over a number of competitors.
