  Large language models (LLMs) can acquire strong code-generation capabilities
through few-shot learning. In contrast, supervised fine-tuning is still needed
for smaller models to achieve good performance. Such fine-tuning demands a
large number of task-specific NL-code pairs, which are expensive to obtain. In
this paper, we attempt to transfer the code generation ability of an LLM to a
smaller model with the aid of weakly-supervised data. More specifically, we
propose explicit knowledge transfer (EKT), which uses the few-shot capabilities
of a teacher LLM to create NL-code pairs that we then filter for correctness
and fine-tune the student on. We evaluate EKT on the task of generating code
solutions to math word problems from the GSM8k dataset. We find that EKT not
only yields better performance than training with expert iteration, but also
outperforms knowledge distillation, another form of knowledge transfer. A
GPT-Neo 1.3B model trained using EKT with a GPT-J teacher achieves a 12.4%
pass@100 on GSM8k, while the same student and teacher trained with knowledge
distillation yield only a 3.7% pass@100. We also show that it is possible for a
student model to outperform the teacher using EKT.
