  Dialogue systems controlled by predefined or rule-based scenarios derived
from counseling techniques, such as cognitive behavioral therapy (CBT), play an
important role in mental health apps. Despite the need for responsible
responses, it is conceivable that using the newly emerging LLMs to generate
contextually relevant utterances will enhance these apps. In this study, we
construct dialogue modules based on a CBT scenario focused on conventional
Socratic questioning using two kinds of LLMs: a Transformer-based dialogue
model further trained with a social media empathetic counseling dataset,
provided by Osaka Prefecture (OsakaED), and GPT-4, a state-of-the art LLM
created by OpenAI. By comparing systems that use LLM-generated responses with
those that do not, we investigate the impact of generated responses on
subjective evaluations such as mood change, cognitive change, and dialogue
quality (e.g., empathy). As a result, no notable improvements are observed when
using the OsakaED model. When using GPT-4, the amount of mood change, empathy,
and other dialogue qualities improve significantly. Results suggest that GPT-4
possesses a high counseling ability. However, they also indicate that even when
using a dialogue model trained with a human counseling dataset, it does not
necessarily yield better outcomes compared to scenario-based dialogues. While
presenting LLM-generated responses, including GPT-4, and having them interact
directly with users in real-life mental health care services may raise ethical
issues, it is still possible for human professionals to produce example
responses or response templates using LLMs in advance in systems that use
rules, scenarios, or example responses.
