  This paper presents an automatic method to evaluate the naturalness of
natural language generation in dialogue systems. While this task was previously
rendered through expensive and time-consuming human labor, we present this
novel task of automatic naturalness evaluation of generated language. By
fine-tuning the BERT model, our proposed naturalness evaluation method shows
robust results and outperforms the baselines: support vector machines,
bi-directional LSTMs, and BLEURT. In addition, the training speed and
evaluation performance of naturalness model are improved by transfer learning
from quality and informativeness linguistic knowledge.
