  Recent diagnostic datasets on compositional generalization, such as SCAN
(Lake and Baroni, 2018) and COGS (Kim and Linzen, 2020), expose severe problems
in models trained from scratch on these datasets. However, in contrast to this
poor performance, state-of-the-art models trained on larger and more general
datasets show better generalization ability. In this work, to reconcile this
inconsistency, we conduct an empirical analysis by training Transformer models
on a variety of training sets with different data factors, including dataset
scale, pattern complexity, example difficulty, etc. First, we show that
increased dataset complexity can lead to better generalization behavior on
multiple different generalization challenges. To further understand this
improvement, we show two axes of the benefit from more complex datasets: they
provide more diverse examples so compositional understanding becomes more
effective, and they also prevent ungeneralizable memorization of the examples
due to reduced example repetition frequency. Finally, we explore how training
examples of different difficulty levels influence generalization differently.
On synthetic datasets, simple examples invoke stronger compositionality than
hard examples do. On larger-scale real language datasets, while hard examples
become more important potentially to ensure decent data coverage, a balanced
mixture of simple and hard examples manages to induce the strongest
generalizability. The code and data for this work are available at
https://github.com/owenzx/data4comp
