  Kaplan et al. [2020] (`Kaplan') and Hoffmann et al. [2022] (`Chinchilla')
studied the scaling behavior of transformers trained on next-token language
prediction. These studies produced different estimates for how the number of
parameters ($N$) and training tokens ($D$) should be set to achieve the lowest
possible loss for a given compute budget ($C$). Kaplan: $N_\text{optimal}
\propto C^{0.73}$, Chinchilla: $N_\text{optimal} \propto C^{0.50}$. This note
finds that much of this discrepancy can be attributed to Kaplan counting
non-embedding rather than total parameters, combined with their analysis being
performed at small scale. Simulating the Chinchilla study under these
conditions produces biased scaling coefficients close to Kaplan's. Hence, this
note reaffirms Chinchilla's scaling coefficients, by explaining the cause of
Kaplan's original overestimation.
