  Humans can understand and produce new utterances effortlessly, thanks to
their compositional skills. Once a person learns the meaning of a new verb
"dax," he or she can immediately understand the meaning of "dax twice" or "sing
and dax." In this paper, we introduce the SCAN domain, consisting of a set of
simple compositional navigation commands paired with the corresponding action
sequences. We then test the zero-shot generalization capabilities of a variety
of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence
methods. We find that RNNs can make successful zero-shot generalizations when
the differences between training and test commands are small, so that they can
apply "mix-and-match" strategies to solve the task. However, when
generalization requires systematic compositional skills (as in the "dax"
example above), RNNs fail spectacularly. We conclude with a proof-of-concept
experiment in neural machine translation, suggesting that lack of systematicity
might be partially responsible for neural networks' notorious training data
thirst.
