  Multimodal signals are more powerful than unimodal data for emotion
recognition since they can represent emotions more comprehensively. In this
paper, we introduce deep canonical correlation analysis (DCCA) to multimodal
emotion recognition. The basic idea behind DCCA is to transform each modality
separately and coordinate different modalities into a hyperspace by using
specified canonical correlation analysis constraints. We evaluate the
performance of DCCA on five multimodal datasets: the SEED, SEED-IV, SEED-V,
DEAP, and DREAMER datasets. Our experimental results demonstrate that DCCA
achieves state-of-the-art recognition accuracy rates on all five datasets:
94.58% on the SEED dataset, 87.45% on the SEED-IV dataset, 84.33% and 85.62%
for two binary classification tasks and 88.51% for a four-category
classification task on the DEAP dataset, 83.08% on the SEED-V dataset, and
88.99%, 90.57%, and 90.67% for three binary classification tasks on the DREAMER
dataset. We also compare the noise robustness of DCCA with that of existing
methods when adding various amounts of noise to the SEED-V dataset. The
experimental results indicate that DCCA has greater robustness. By visualizing
feature distributions with t-SNE and calculating the mutual information between
different modalities before and after using DCCA, we find that the features
transformed by DCCA from different modalities are more homogeneous and
discriminative across emotions.
