  Lowering the precision of neural networks from the prevalent 32-bit precision
has long been considered harmful to performance, despite the gain in space and
time. Many works propose various techniques to implement half-precision neural
networks, but none study pure 16-bit settings. This paper investigates the
unexpected performance gain of pure 16-bit neural networks over the 32-bit
networks in classification tasks. We present extensive experimental results
that favorably compare various 16-bit neural networks' performance to those of
the 32-bit models. In addition, a theoretical analysis of the efficiency of
16-bit models is provided, which is coupled with empirical evidence to back it
up. Finally, we discuss situations in which low-precision training is indeed
detrimental.
