  Language models have been shown to exhibit positive scaling, where
performance improves as models are scaled up in terms of size, compute, or
data. In this work, we introduce NeQA, a dataset consisting of questions with
negation in which language models do not exhibit straightforward positive
scaling. We show that this task can exhibit inverse scaling, U-shaped scaling,
or positive scaling, and the three scaling trends shift in this order as we use
more powerful prompting methods or model families. We hypothesize that solving
NeQA depends on two subtasks: question answering (task 1) and negation
understanding (task 2). We find that task 1 has linear scaling, while task 2
has sigmoid-shaped scaling with an emergent transition point, and composing
these two scaling trends yields the final scaling trend of NeQA. Our work
reveals and provides a way to analyze the complex scaling trends of language
models.
