  In contemporary society, the issue of psychological health has become
increasingly prominent, characterized by the diversification, complexity, and
universality of mental disorders. Cognitive Behavioral Therapy (CBT), currently
the most influential and clinically effective psychological treatment method
with no side effects, has limited coverage and poor quality in most countries.
In recent years, researches on the recognition and intervention of emotional
disorders using large language models (LLMs) have been validated, providing new
possibilities for psychological assistance therapy. However, are LLMs truly
possible to conduct cognitive behavioral therapy? Many concerns have been
raised by mental health experts regarding the use of LLMs for therapy. Seeking
to answer this question, we collected real CBT corpus from online video
websites, designed and conducted a targeted automatic evaluation framework
involving the evaluation of emotion tendency of generated text, structured
dialogue pattern and proactive inquiry ability. For emotion tendency, we
calculate the emotion tendency score of the CBT dialogue text generated by each
model. For structured dialogue pattern, we use a diverse range of automatic
evaluation metrics to compare speaking style, the ability to maintain
consistency of topic and the use of technology in CBT between different models
. As for inquiring to guide the patient, we utilize PQA (Proactive Questioning
Ability) metric. We also evaluated the CBT ability of the LLM after integrating
a CBT knowledge base to explore the help of introducing additional knowledge to
enhance the model's CBT counseling ability. Four LLM variants with excellent
performance on natural language processing are evaluated, and the experimental
result shows the great potential of LLMs in psychological counseling realm,
especially after combining with other technological means.
