  Recent work has advocated for training AI models on ever-larger datasets,
arguing that as the size of a dataset increases, the performance of a model
trained on that dataset will correspondingly increase (referred to as "scaling
laws"). In this paper, we draw on literature from the social sciences and
machine learning to critically interrogate these claims. We argue that this
scaling law relationship depends on metrics used to measure performance that
may not correspond with how different groups of people perceive the quality of
models' output. As the size of datasets used to train large AI models grows and
AI systems impact ever larger groups of people, the number of distinct
communities represented in training or evaluation datasets grows. It is thus
even more likely that communities represented in datasets may have values or
preferences not reflected in (or at odds with) the metrics used to evaluate
model performance in scaling laws. Different communities may also have values
in tension with each other, leading to difficult, potentially irreconcilable
choices about metrics used for model evaluations -- threatening the validity of
claims that model performance is improving at scale. We end the paper with
implications for AI development: that the motivation for scraping ever-larger
datasets may be based on fundamentally flawed assumptions about model
performance. That is, models may not, in fact, continue to improve as the
datasets get larger -- at least not for all people or communities impacted by
those models. We suggest opportunities for the field to rethink norms and
values in AI development, resisting claims for universality of large models,
fostering more local, small-scale designs, and other ways to resist the impetus
towards scale in AI.
