  State-of-the-art generic low-precision training algorithms use a mix of
16-bit and 32-bit precision, creating the folklore that 16-bit hardware compute
units alone are not enough to maximize model accuracy. As a result, deep
learning accelerators are forced to support both 16-bit and 32-bit
floating-point units (FPUs), which is more costly than only using 16-bit FPUs
for hardware design. We ask: can we train deep learning models only with 16-bit
floating-point units, while still matching the model accuracy attained by
32-bit training? Towards this end, we study 16-bit-FPU training on the widely
adopted BFloat16 unit. While these units conventionally use nearest rounding to
cast output to 16-bit precision, we show that nearest rounding for model weight
updates often cancels small updates, which degrades the convergence and model
accuracy. Motivated by this, we study two simple techniques well-established in
numerical analysis, stochastic rounding and Kahan summation, to remedy the
model accuracy degradation in 16-bit-FPU training. We demonstrate that these
two techniques can enable up to 7% absolute validation accuracy gain in
16-bit-FPU training. This leads to 0.1% lower to 0.2% higher validation
accuracy compared to 32-bit training across seven deep learning applications.
