  This paper introduces EmpathyEar, a pioneering open-source, avatar-based
multimodal empathetic chatbot, to fill the gap in traditional text-only
empathetic response generation (ERG) systems. Leveraging the advancements of a
large language model, combined with multimodal encoders and generators,
EmpathyEar supports user inputs in any combination of text, sound, and vision,
and produces multimodal empathetic responses, offering users, not just textual
responses but also digital avatars with talking faces and synchronized
speeches. A series of emotion-aware instruction-tuning is performed for
comprehensive emotional understanding and generation capabilities. In this way,
EmpathyEar provides users with responses that achieve a deeper emotional
resonance, closely emulating human-like empathy. The system paves the way for
the next emotional intelligence, for which we open-source the code for public
access.
