  Mental health care poses an increasingly serious challenge to modern
societies. In this context, there has been a surge in research that utilizes
information technologies to address mental health problems, including those
aiming to develop counseling dialogue systems. However, there is a need for
more evaluations of the performance of counseling dialogue systems that use
large language models. For this study, we collected counseling dialogue data
via role-playing scenarios involving expert counselors, and the utterances were
annotated with the intentions of the counselors. To determine the feasibility
of a dialogue system in real-world counseling scenarios, third-party counselors
evaluated the appropriateness of responses from human counselors and those
generated by GPT-4 in identical contexts in role-play dialogue data. Analysis
of the evaluation results showed that the responses generated by GPT-4 were
competitive with those of human counselors.
