  The integration of conversational artificial intelligence (AI) into mental
health care promises a new horizon for therapist-client interactions, aiming to
closely emulate the depth and nuance of human conversations. Despite the
potential, the current landscape of conversational AI is markedly limited by
its reliance on single-modal data, constraining the systems' ability to
empathize and provide effective emotional support. This limitation stems from a
paucity of resources that encapsulate the multimodal nature of human
communication essential for therapeutic counseling. To address this gap, we
introduce the Multimodal Emotional Support Conversation (MESC) dataset, a
first-of-its-kind resource enriched with comprehensive annotations across text,
audio, and video modalities. This dataset captures the intricate interplay of
user emotions, system strategies, system emotion, and system responses, setting
a new precedent in the field. Leveraging the MESC dataset, we propose a general
Sequential Multimodal Emotional Support framework (SMES) grounded in
Therapeutic Skills Theory. Tailored for multimodal dialogue systems, the SMES
framework incorporates an LLM-based reasoning model that sequentially generates
user emotion recognition, system strategy prediction, system emotion
prediction, and response generation. Our rigorous evaluations demonstrate that
this framework significantly enhances the capability of AI systems to mimic
therapist behaviors with heightened empathy and strategic responsiveness. By
integrating multimodal data in this innovative manner, we bridge the critical
gap between emotion recognition and emotional support, marking a significant
advancement in conversational AI for mental health support.
