  While integer arithmetic has been widely adopted for improved performance in
deep quantized neural network inference, training remains a task primarily
executed using floating point arithmetic. This is because both high dynamic
range and numerical accuracy are central to the success of most modern training
algorithms. However, due to its potential for computational, storage and energy
advantages in hardware accelerators, neural network training methods that can
be implemented with low precision integer-only arithmetic remains an active
research challenge. In this paper, we present NITI, an efficient deep neural
network training framework that stores all parameters and intermediate values
as integers, and computes exclusively with integer arithmetic. A pseudo
stochastic rounding scheme that eliminates the need for external random number
generation is proposed to facilitate conversion from wider intermediate results
to low precision storage. Furthermore, a cross-entropy loss backpropagation
scheme computed with integer-only arithmetic is proposed. A proof-of-concept
open-source software implementation of NITI that utilizes native 8-bit integer
operations in modern GPUs to achieve end-to-end training is presented. When
compared with an equivalent training setup implemented with floating point
storage and arithmetic, NITI achieves negligible accuracy degradation on the
MNIST and CIFAR10 datasets using 8-bit integer storage and computation. On
ImageNet, 16-bit integers are needed for weight accumulation with an 8-bit
datapath. This achieves training results comparable to all-floating-point
implementations.
