  Predictable behavior from scaling advanced AI systems is an extremely
desirable property. Although a well-established literature exists on how
pretraining performance scales, the literature on how particular downstream
capabilities scale is significantly muddier. In this work, we take a step back
and ask: why has predicting specific downstream capabilities with scale
remained elusive? While many factors are certainly responsible, we identify a
new factor that makes modeling scaling behavior on widely used multiple-choice
question-answering benchmarks challenging. Using five model families and twelve
well-established multiple-choice benchmarks, we show that downstream
performance is computed from negative log likelihoods via a sequence of
transformations that progressively degrade the statistical relationship between
performance and scale. We then reveal the mechanism causing this degradation:
downstream metrics require comparing the correct choice against a small number
of specific incorrect choices, meaning accurately predicting downstream
capabilities requires predicting not just how probability mass concentrates on
the correct choice with scale, but also how probability mass fluctuates on
specific incorrect choices with scale. We empirically study how probability
mass on the correct choice co-varies with probability mass on incorrect choices
with increasing compute, suggesting that scaling laws for incorrect choices
might be achievable. Our work also explains why pretraining scaling laws are
commonly regarded as more predictable than downstream capabilities and
contributes towards establishing scaling-predictable evaluations of frontier AI
models.
