  Agents represent one of the most emerging applications of Large Language
Models (LLMs) and Generative AI, with their effectiveness hinging on multimodal
capabilities to navigate complex user environments. Conversational Health
Agents (CHAs), a prime example of this, are redefining healthcare by offering
nuanced support that transcends textual analysis to incorporate emotional
intelligence. This paper introduces an LLM-based CHA engineered for rich,
multimodal dialogue-especially in the realm of mental health support. It
adeptly interprets and responds to users' emotional states by analyzing
multimodal cues, thus delivering contextually aware and empathetically resonant
verbal responses. Our implementation leverages the versatile openCHA framework,
and our comprehensive evaluation involves neutral prompts expressed in diverse
emotional tones: sadness, anger, and joy. We evaluate the consistency and
repeatability of the planning capability of the proposed CHA. Furthermore,
human evaluators critique the CHA's empathic delivery, with findings revealing
a striking concordance between the CHA's outputs and evaluators' assessments.
These results affirm the indispensable role of vocal (soon multimodal) emotion
recognition in strengthening the empathetic connection built by CHAs, cementing
their place at the forefront of interactive, compassionate digital health
solutions.
