  We study trends in model size of notable machine learning systems over time
using a curated dataset. From 1950 to 2018, model size in language models
increased steadily by seven orders of magnitude. The trend then accelerated,
with model size increasing by another five orders of magnitude in just 4 years
from 2018 to 2022. Vision models grew at a more constant pace, totaling 7
orders of magnitude of growth between 1950 and 2022.
  We also identify that, since 2020, there have been many language models below
20B parameters, many models above 70B parameters, but a scarcity of models in
the 20-70B parameter range. We refer to that scarcity as the parameter gap.
  We provide some stylized facts about the parameter gap and propose a few
hypotheses to explain it. The explanations we favor are: (a) increasing model
size beyond 20B parameters requires adopting different parallelism techniques,
which makes mid-sized models less cost-effective, (b) GPT-3 was one order of
magnitude larger than previous language models, and researchers afterwards
primarily experimented with bigger models to outperform it. While these
dynamics likely exist, and we believe they play some role in generating the
gap, we don't have high confidence that there are no other, more important
dynamics at play.
