  We study recent research advances that improve large language models through
efficient pre-training and scaling, and open datasets and tools. We combine
these advances to introduce Cerebras-GPT, a family of open compute-optimal
language models scaled from 111M to 13B parameters. We train Cerebras-GPT
models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules
for efficient pre-training (highest accuracy for a given compute budget). We
characterize the predictable power-law scaling and compare Cerebras-GPT with
other publicly-available models to show all Cerebras-GPT models have
state-of-the-art training efficiency on both pre-training and downstream
objectives. We describe our learnings including how Maximal Update
Parameterization ($\mu$P) can further improve large model scaling, improving
accuracy and hyperparameter predictability at scale. We release our pre-trained
models and code, making this paper the first open and reproducible work
comparing compute-optimal model scaling to models trained on fixed dataset
sizes. Cerebras-GPT models are available on HuggingFace:
https://huggingface.co/cerebras.
