  Sequence-to-sequence (seq2seq) learners are widely used, but we still have
only limited knowledge about what inductive biases shape the way they
generalize. We address that by investigating how popular seq2seq learners
generalize in tasks that have high ambiguity in the training data. We use SCAN
and three new tasks to study learners' preferences for memorization,
arithmetic, hierarchical, and compositional reasoning. Further, we connect to
Solomonoff's theory of induction and propose to use description length as a
principled and sensitive measure of inductive biases.
  In our experimental study, we find that LSTM-based learners can learn to
perform counting, addition, and multiplication by a constant from a single
training example. Furthermore, Transformer and LSTM-based learners show a bias
toward the hierarchical induction over the linear one, while CNN-based learners
prefer the opposite. On the SCAN dataset, we find that CNN-based, and, to a
lesser degree, Transformer- and LSTM-based learners have a preference for
compositional generalization over memorization. Finally, across all our
experiments, description length proved to be a sensitive measure of inductive
biases.
