Improving Tree-Structured Decoder Training for Code Generation via
  Mutual Learning
  Code generation aims to automatically generate a piece of code given an input
natural language utterance. Currently, among dominant models, it is treated as
a sequence-to-tree task, where a decoder outputs a sequence of actions
corresponding to the pre-order traversal of an Abstract Syntax Tree. However,
such a decoder only exploits the preorder traversal based preceding actions,
which are insufficient to ensure correct action predictions. In this paper, we
first throughly analyze the context modeling difference between neural code
generation models with different traversals based decodings (preorder traversal
vs breadth-first traversal), and then propose to introduce a mutual learning
framework to jointly train these models. Under this framework, we continuously
enhance both two models via mutual distillation, which involves synchronous
executions of two one-to-one knowledge transfers at each training step. More
specifically, we alternately choose one model as the student and the other as
its teacher, and require the student to fit the training data and the action
prediction distributions of its teacher. By doing so, both models can fully
absorb the knowledge from each other and thus could be improved simultaneously.
Experimental results and in-depth analysis on several benchmark datasets
demonstrate the effectiveness of our approach. We release our code at
https://github.com/DeepLearnXMU/CGML.
