  Automatic dialogue response evaluator has been proposed as an alternative to
automated metrics and human evaluation. However, existing automatic evaluators
achieve only moderate correlation with human judgement and they are not robust.
In this work, we propose to build a reference-free evaluator and exploit the
power of semi-supervised training and pretrained (masked) language models.
Experimental results demonstrate that the proposed evaluator achieves a strong
correlation (> 0.6) with human judgement and generalizes robustly to diverse
responses and corpora. We open-source the code and data in
https://github.com/ZHAOTING/dialog-processing.
