  Neuromorphic Multiply-And-Accumulate (MAC) circuits utilizing synaptic weight
elements based on SRAM or novel Non-Volatile Memories (NVMs) provide a
promising approach for highly efficient hardware representations of neural
networks. NVM density and robustness requirements suggest that off-line
training is the right choice for "edge" devices, since the requirements for
synapse precision are much less stringent. However, off-line training using
ideal mathematical weights and activations can result in significant loss of
inference accuracy when applied to non-ideal hardware. Non-idealities such as
multi-bit quantization of weights and activations, non-linearity of weights,
finite max/min ratios of NVM elements, and asymmetry of positive and negative
weight components all result in degraded inference accuracy. In this work, it
is demonstrated that non-ideal Multi-Layer Perceptron (MLP) architectures using
low bitwidth weights and activations can be trained with negligible loss of
inference accuracy relative to their Floating Point-trained counterparts using
a proposed off-line, continuously differentiable HW-aware training algorithm.
The proposed algorithm is applicable to a wide range of hardware models, and
uses only standard neural network training methods. The algorithm is
demonstrated on the MNIST and EMNIST datasets, using standard MLPs.
