  Low precision operations can provide scalability, memory savings,
portability, and energy efficiency. This paper proposes SWALP, an approach to
low precision training that averages low-precision SGD iterates with a modified
learning rate schedule. SWALP is easy to implement and can match the
performance of full-precision SGD even with all numbers quantized down to 8
bits, including the gradient accumulators. Additionally, we show that SWALP
converges arbitrarily close to the optimal solution for quadratic objectives,
and to a noise ball asymptotically smaller than low precision SGD in strongly
convex settings.
