  Robust and effective scaling of models from small to large width typically
requires the precise adjustment of many algorithmic and architectural details,
such as parameterization and optimizer choices. In this work, we propose a new
perspective on parameterization by investigating a key assumption in prior work
about the alignment between parameters and data and derive new theoretical
results under weaker assumptions and a broader set of optimizers. Our extensive
empirical investigation includes tens of thousands of models trained with all
combinations of three optimizers, four parameterizations, several alignment
assumptions, more than a dozen learning rates, and fourteen model sizes up to
26.8B parameters. We find that the best learning rate scaling prescription
would often have been excluded by the assumptions in prior work. Our results
show that all parameterizations, not just maximal update parameterization
(muP), can achieve hyperparameter transfer; moreover, our novel per-layer
learning rate prescription for standard parameterization outperforms muP.
Finally, we demonstrate that an overlooked aspect of parameterization, the
epsilon parameter in Adam, must be scaled correctly to avoid gradient underflow
and propose Adam-atan2, a new numerically stable, scale-invariant version of
Adam that eliminates the epsilon hyperparameter entirely.
