  Automatic code generation is to generate the program code according to the
given natural language description. The current mainstream approach uses neural
networks to encode natural language descriptions, and output abstract syntax
trees (AST) at the decoder, then convert the AST into program code. While the
generated code largely conforms to specific syntax rules, two problems are
still ignored. One is missing program testing, an essential step in the process
of complete code implementation; the other is only focusing on the syntax
compliance of the generated code, while ignoring the more important program
functional requirements. The paper proposes a CodeGen-Test model, which adds
program testing steps and incorporates program testing information to
iteratively generate code that meets the functional requirements of the
program, thereby improving the quality of code generation. At the same time,
the paper proposes a new evaluation metric, test accuracy (Test-Acc), which
represents the proportion of passing program test in generated code. Different
from the previous evaluation metric, which only evaluates the quality of code
generation from the perspective of character similarity, the Test-Acc can
evaluate the quality of code generation from the Program functions. Moreover,
the paper evaluates the CodeGen-test model on a python data set "hearthstone
legend". The experimental results show the proposed method can effectively
improve the quality of generated code. Compared with the existing optimal
model, CodeGen-Test model improves the Bleu value by 0.2%, Rouge-L value by
0.3% and Test-Acc by 6%.
