  Automatically evaluating the quality of responses in open-domain dialogue
systems is a challenging but crucial task. Current evaluation metrics often
fail to align with human judgments, especially when assessing responses that
are grammatically correct. To address this issue, we propose a novel metric,
called CausalScore, which assesses the relevance of responses by measuring the
causal strength between dialogue histories and responses. The causal strength
is estimated by utilizing both unconditional dependence and conditional
dependencies from the dialogue history to responses. We compare our metric with
the existing competitive metrics in terms of their alignment with human
judgements. Our experimental results demonstrate that CausalScore significantly
surpasses existing state-of-the-art metrics by aligning better with human
judgements. Additionally, we collect a new dialogue dataset CGDIALOG+ with
human-annotated causal relations and a set of pairwise human judgements to
facilitate the development of future automatic metrics.
