  Scale has been a major driving force in improving machine learning
performance, and understanding scaling laws is essential for strategic planning
for a sustainable model quality performance growth, long-term resource planning
and developing efficient system infrastructures to support large-scale models.
In this paper, we study empirical scaling laws for DLRM style recommendation
models, in particular Click-Through Rate (CTR). We observe that model quality
scales with power law plus constant in model size, data size and amount of
compute used for training. We characterize scaling efficiency along three
different resource dimensions, namely data, parameters and compute by comparing
the different scaling schemes along these axes. We show that parameter scaling
is out of steam for the model architecture under study, and until a
higher-performing model architecture emerges, data scaling is the path forward.
The key research questions addressed by this study include: Does a
recommendation model scale sustainably as predicted by the scaling laws? Or are
we far off from the scaling law predictions? What are the limits of scaling?
What are the implications of the scaling laws on long-term hardware/system
development?
