  While Deep Neural Networks (DNNs) push the state-of-the-art in many machine
learning applications, they often require millions of expensive floating-point
operations for each input classification. This computation overhead limits the
applicability of DNNs to low-power, embedded platforms and incurs high cost in
data centers. This motivates recent interests in designing low-power,
low-latency DNNs based on fixed-point, ternary, or even binary data precision.
While recent works in this area offer promising results, they often lead to
large accuracy drops when compared to the floating-point networks. We propose a
novel approach to map floating-point based DNNs to 8-bit dynamic fixed-point
networks with integer power-of-two weights with no change in network
architecture. Our dynamic fixed-point DNNs allow different radix points between
layers. During inference, power-of-two weights allow multiplications to be
replaced with arithmetic shifts, while the 8-bit fixed-point representation
simplifies both the buffer and adder design. In addition, we propose a hardware
accelerator design to achieve low-power, low-latency inference with
insignificant degradation in accuracy. Using our custom accelerator design with
the CIFAR-10 and ImageNet datasets, we show that our method achieves
significant power and energy savings while increasing the classification
accuracy.
