  Depression recognition based on physiological signals such as functional
near-infrared spectroscopy (fNIRS) and electroencephalogram (EEG) has made
considerable progress. However, most existing studies ignore the
complementarity and semantic consistency of multimodal physiological signals
under the same stimulation task in complex spatio-temporal patterns. In this
paper, we introduce a multimodal physiological signals representation learning
framework using Siamese architecture via multiscale contrasting for depression
recognition (MRLMC). First, fNIRS and EEG are transformed into different but
correlated data based on a time-domain data augmentation strategy. Then, we
design a spatio-temporal contrasting module to learn the representation of
fNIRS and EEG through weight-sharing multiscale spatio-temporal convolution.
Furthermore, to enhance the learning of semantic representation associated with
stimulation tasks, a semantic consistency contrast module is proposed, aiming
to maximize the semantic similarity of fNIRS and EEG. Extensive experiments on
publicly available and self-collected multimodal physiological signals datasets
indicate that MRLMC outperforms the state-of-the-art models. Moreover, our
proposed framework is capable of transferring to multimodal time series
downstream tasks.
