  The massive computational costs associated with large language model (LLM)
pretraining have spurred great interest in reduced-precision floating-point
representations to accelerate the process. As a result, the BrainFloat16 (BF16)
precision has become the de facto standard for LLM training, with hardware
support included in recent accelerators. This trend has gone even further in
the latest processors, where FP8 has recently been introduced. However, prior
experience with FP16, which was found to be less stable than BF16, raises
concerns as to whether FP8, with even fewer bits than FP16, can be a
cost-effective option for LLM training. We argue that reduced-precision
training schemes must have similar training stability and hyperparameter
sensitivities to their higher-precision counterparts in order to be
cost-effective. However, we find that currently available methods for FP8
training are not robust enough to allow their use as economical replacements.
This prompts us to investigate the stability of reduced-precision LLM training
in terms of robustness across random seeds and learning rates. To this end, we
propose new evaluation techniques and a new metric for quantifying loss
landscape sharpness in autoregressive language models. By simulating
incremental bit reductions in floating-point representations, we analyze the
relationship between representational power and training stability with the
intent of aiding future research into the field.
