  Natural Language Generation (NLG) refers to the operation of expressing the
calculation results of a system in human language. Since the quality of
generated sentences from an NLG model cannot be fully represented using only
quantitative evaluation, they are evaluated using qualitative evaluation by
humans in which the meaning or grammar of a sentence is scored according to a
subjective criterion. Nevertheless, the existing evaluation methods have a
problem as a large score deviation occurs depending on the criteria of
evaluators. In this paper, we propose Grammar Accuracy Evaluation (GAE) that
can provide the specific evaluating criteria. As a result of analyzing the
quality of machine translation by BLEU and GAE, it was confirmed that the BLEU
score does not represent the absolute performance of machine translation models
and GAE compensates for the shortcomings of BLEU with flexible evaluation of
alternative synonyms and changes in sentence structure.
