  Evaluating Natural Language Generation (NLG) systems is a challenging task.
Firstly, the metric should ensure that the generated hypothesis reflects the
reference's semantics. Secondly, it should consider the grammatical quality of
the generated sentence. Thirdly, it should be robust enough to handle various
surface forms of the generated sentence. Thus, an effective evaluation metric
has to be multifaceted. In this paper, we propose an automatic evaluation
metric incorporating several core aspects of natural language understanding
(language competence, syntactic and semantic variation). Our proposed metric,
RoMe, is trained on language features such as semantic similarity combined with
tree edit distance and grammatical acceptability, using a self-supervised
neural network to assess the overall quality of the generated sentence.
Moreover, we perform an extensive robustness analysis of the state-of-the-art
methods and RoMe. Empirical results suggest that RoMe has a stronger
correlation to human judgment over state-of-the-art metrics in evaluating
system-generated sentences across several NLG tasks.
