  Despite their strong performance on many tasks, pre-trained language models
have been shown to struggle on out-of-distribution compositional
generalization. Meanwhile, recent work has shown considerable improvements on
many NLP tasks from model scaling. Can scaling up model size also improve
compositional generalization in semantic parsing? We evaluate encoder-decoder
models up to 11B parameters and decoder-only models up to 540B parameters, and
compare model scaling curves for three different methods for applying a
pre-trained language model to a new task: fine-tuning all parameters, prompt
tuning, and in-context learning. We observe that fine-tuning generally has flat
or negative scaling curves on out-of-distribution compositional generalization
in semantic parsing evaluations. In-context learning has positive scaling
curves, but is generally outperformed by much smaller fine-tuned models.
Prompt-tuning can outperform fine-tuning, suggesting further potential
improvements from scaling as it exhibits a more positive scaling curve.
Additionally, we identify several error trends that vary with model scale. For
example, larger models are generally better at modeling the syntax of the
output space, but are also more prone to certain types of overfitting. Overall,
our study highlights limitations of current techniques for effectively
leveraging model scale for compositional generalization, while our analysis
also suggests promising directions for future work.
