  Systems for multimodal emotion recognition (ER) are commonly trained to
extract features from different modalities (e.g., visual, audio, and textual)
that are combined to predict individual basic emotions. However, compound
emotions often occur in real-world scenarios, and the uncertainty of
recognizing such complex emotions over diverse modalities is challenging for
feature-based models As an alternative, emerging multimodal large language
models (LLMs) like BERT and LLaMA rely on explicit non-verbal cues that may be
translated from different non-textual modalities (e.g., audio and visual) into
text. Textualization of modalities augments data with emotional cues to help
the LLM encode the interconnections between all modalities in a shared text
space. In such text-based models, prior knowledge of ER tasks is leveraged to
textualize relevant nonverbal cues such as audio tone from vocal expressions,
and action unit intensity from facial expressions. Since the pre-trained
weights are publicly available for many LLMs, training on large-scale datasets
is unnecessary, allowing fine-tuning for downstream tasks such as compound ER
(CER). This paper compares the potential of text- and feature-based approaches
for compound multimodal ER in videos. Experiments were conducted on the
challenging C-EXPR-DB dataset in the wild for CER, and contrasted with results
on the MELD dataset for basic ER. Our results indicate that multimodal
textualization provides lower accuracy than feature-based models on C-EXPR-DB,
where text transcripts are captured in the wild. However, higher accuracy can
be achieved when the video data has rich transcripts. Our code is available.
