  Recently, there is a surge of interest in applying pre-trained language
models (Pr-LM) in automatic open-domain dialog evaluation. Pr-LMs offer a
promising direction for addressing the multi-domain evaluation challenge. Yet,
the impact of different Pr-LMs on the performance of automatic metrics is not
well-understood. This paper examines 8 different Pr-LMs and studies their
impact on three typical automatic dialog evaluation metrics across three
different dialog evaluation benchmarks. Specifically, we analyze how the choice
of Pr-LMs affects the performance of automatic metrics. Extensive correlation
analyses on each of the metrics are performed to assess the effects of
different Pr-LMs along various axes, including pre-training objectives, dialog
evaluation criteria, model size, and cross-dataset robustness. This study
serves as the first comprehensive assessment of the effects of different Pr-LMs
on automatic dialog evaluation.
