  Multimodal emotion recognition techniques are increasingly essential for
assessing mental states. Image-based methods, however, tend to focus
predominantly on overt visual cues and often overlook subtler mental state
changes. Psychophysiological research has demonstrated that HR and skin
temperature are effective in detecting ANS activities, thereby revealing these
subtle changes. However, traditional HR tools are generally more costly and
less portable, while skin temperature analysis usually necessitates extensive
manual processing. Advances in remote-PPG and automatic thermal ROI detection
algorithms have been developed to address these issues, yet their accuracy in
practical applications remains limited. This study aims to bridge this gap by
integrating r-PPG with thermal imaging to enhance prediction performance.
Ninety participants completed a 20-minute questionnaire to induce cognitive
stress, followed by watching a film aimed at eliciting moral elevation. The
results demonstrate that the combination of r-PPG and thermal imaging
effectively detects emotional shifts. Using r-PPG alone, the prediction
accuracy was 77% for cognitive stress and 61% for moral elevation, as
determined by SVM. Thermal imaging alone achieved 79% accuracy for cognitive
stress and 78% for moral elevation, utilizing a RF algorithm. An early fusion
strategy of these modalities significantly improved accuracies, achieving 87%
for cognitive stress and 83% for moral elevation using RF. Further analysis,
which utilized statistical metrics and explainable machine learning methods
including SHAP, highlighted key features and clarified the relationship between
cardiac responses and facial temperature variations. Notably, it was observed
that cardiovascular features derived from r-PPG models had a more pronounced
influence in data fusion, despite thermal imaging's higher predictive accuracy
in unimodal analysis.
