  Large language model pre-training has become increasingly expensive, with
most practitioners relying on scaling laws to allocate compute budgets for
model size and training tokens, commonly referred to as Compute-Optimal or
Chinchilla Optimal. In this paper, we hypothesize a new scaling law that
suggests model performance depends mostly on the amount of compute spent for
transformer-based models, independent of the specific allocation to model size
and dataset size. Using this unified scaling law, we predict that (a) for
inference efficiency, training should prioritize smaller model sizes and larger
training datasets, and (b) assuming the exhaustion of available web datasets,
scaling the model size might be the only way to further improve model
performance.
