  Although empathic interaction between counselor and client is fundamental to
success in the psychotherapeutic process, there are currently few datasets to
aid a computational approach to empathy understanding. In this paper, we
construct a multimodal empathy dataset collected from face-to-face
psychological counseling sessions. The dataset consists of 771 video clips. We
also propose three labels (i.e., expression of experience, emotional reaction,
and cognitive reaction) to describe the degree of empathy between counselors
and their clients. Expression of experience describes whether the client has
expressed experiences that can trigger empathy, and emotional and cognitive
reactions indicate the counselor's empathic reactions. As an elementary
assessment of the usability of the constructed multimodal empathy dataset, an
interrater reliability analysis of annotators' subjective evaluations for video
clips is conducted using the intraclass correlation coefficient and Fleiss'
Kappa. Results prove that our data annotation is reliable. Furthermore, we
conduct empathy prediction using three typical methods, including the tensor
fusion network, the sentimental words aware fusion network, and a simple
concatenation model. The experimental results show that empathy can be well
predicted on our dataset. Our dataset is available for research purposes.
