  Emotional Support Conversation (ESC) is a typical dialogue that can
effectively assist the user in mitigating emotional pressures. However, owing
to the inherent subjectivity involved in analyzing emotions, current
non-artificial methodologies face challenges in effectively appraising the
emotional support capability. These metrics exhibit a low correlation with
human judgments. Concurrently, manual evaluation methods extremely will cause
high costs. To solve these problems, we propose a novel model FEEL (Framework
for Evaluating Emotional Support Capability with Large Lan-guage Models),
employing Large Language Models (LLMs) as evaluators to assess emotional
support capabilities. The model meticulously considers various evaluative
aspects of ESC to apply a more comprehensive and accurate evaluation method for
ESC. Additionally, it employs a probability distribution approach for a more
stable result and integrates an ensemble learning strategy, leveraging multiple
LLMs with assigned weights to enhance evaluation accuracy. To appraise the
performance of FEEL, we conduct extensive experiments on existing ESC model
dialogues. Experimental results demonstrate our model exhibits a substantial
enhancement in alignment with human evaluations compared to the baselines. Our
source code is available at https://github.com/Ansisy/FEEL.
