  Out-of-distribution generalization capabilities of sequence-to-sequence
models can be studied from the lens of two crucial forms of generalization:
length generalization -- the ability to generalize to longer sequences than
ones seen during training, and compositional generalization: the ability to
generalize to token combinations not seen during training. In this work, we
provide first provable guarantees on length and compositional generalization
for common sequence-to-sequence models -- deep sets, transformers, state space
models, and recurrent neural nets -- trained to minimize the prediction error.
Taking a first principles perspective, we study the realizable case, i.e., the
labeling function is realizable on the architecture. We show that limited
capacity versions of these different architectures achieve both length and
compositional generalization. Across different architectures, we also find that
a linear relationship between the learned representation and the representation
in the labeling function is necessary for length and compositional
generalization.
