  In recent years, language models have drastically grown in size, and the
abilities of these models have been shown to improve with scale. The majority
of recent scaling laws studies focused on high-compute high-parameter count
settings, leaving the question of when these abilities begin to emerge largely
unanswered. In this paper, we investigate whether the effects of pre-training
can be observed when the problem size is reduced, modeling a smaller,
reduced-vocabulary language. We show the benefits of pre-training with masked
language modeling (MLM) objective in models as small as 1.25M parameters, and
establish a strong correlation between pre-training perplexity and downstream
performance (GLUE benchmark). We examine downscaling effects, extending scaling
laws to models as small as ~1M parameters. At this scale, we observe a break of
the power law for compute-optimal models and show that the MLM loss does not
scale smoothly with compute-cost (FLOPs) below $2.2 \times 10^{15}$ FLOPs. We
also find that adding layers does not always benefit downstream performance.
