  Recent advancements in reference-free learned metrics for open-domain
dialogue evaluation have been driven by the progress in pre-trained language
models and the availability of dialogue data with high-quality human
annotations. However, current studies predominantly concentrate on English
dialogues, and the generalization of these metrics to other languages has not
been fully examined. This is largely due to the absence of a multilingual
dialogue evaluation benchmark. To address the issue, we introduce xDial-Eval,
built on top of open-source English dialogue evaluation datasets. xDial-Eval
includes 12 turn-level and 6 dialogue-level English datasets, comprising 14930
annotated turns and 8691 annotated dialogues respectively. The English dialogue
data are extended to nine other languages with commercial machine translation
systems. On xDial-Eval, we conduct comprehensive analyses of previous
BERT-based metrics and the recently-emerged large language models. Lastly, we
establish strong self-supervised and multilingual baselines. In terms of
average Pearson correlations over all datasets and languages, the best baseline
outperforms OpenAI's ChatGPT by absolute improvements of 6.5% and 4.6% at the
turn and dialogue levels respectively, albeit with much fewer parameters. The
data and code are publicly available at https://github.com/e0397123/xDial-Eval.
