  Evaluation metrics play a vital role in the growth of an area as it defines
the standard of distinguishing between good and bad models. In the area of code
synthesis, the commonly used evaluation metric is BLEU or perfect accuracy, but
they are not suitable enough to evaluate codes, because BLEU is originally
designed to evaluate the natural language, neglecting important syntactic and
semantic features of codes, and perfect accuracy is too strict thus it
underestimates different outputs with the same semantic logic. To remedy this,
we introduce a new automatic evaluation metric, dubbed CodeBLEU. It absorbs the
strength of BLEU in the n-gram match and further injects code syntax via
abstract syntax trees (AST) and code semantics via data-flow. We conduct
experiments by evaluating the correlation coefficient between CodeBLEU and
quality scores assigned by the programmers on three code synthesis tasks, i.e.,
text-to-code, code translation, and code refinement. Experimental results show
that our proposed CodeBLEU can achieve a better correlation with programmer
assigned scores compared with BLEU and accuracy.
