  Convolutional neural networks (CNNs) achieve state-of-the-art performance at
the cost of becoming deeper and larger. Although quantization (both fixed-point
and floating-point) has proven effective for reducing storage and memory
access, two challenges -- 1) accuracy loss caused by quantization without
calibration, fine-tuning or re-training for deep CNNs and 2) hardware
inefficiency caused by floating-point quantization -- prevent processors from
completely leveraging the benefits.
  In this paper, we propose a low-precision floating-point quantization
oriented processor, named Phoenix, to address the above challenges. We
primarily have three key observations: 1) 8-bit floating-point quantization
incurs less error than 8-bit fixed-point quantization; 2) without using any
calibration, fine-tuning or re-training techniques, normalization before
quantization further reduces accuracy degradation; 3) 8-bit floating-point
multiplier achieves higher hardware efficiency than 8-bit fixed-point
multiplier if the full-precision product is applied. Based on these key
observations, we propose a normalization-oriented 8-bit floating-point
quantization method to reduce storage and memory access with negligible
accuracy loss (within 0.5%/0.3% for top-1/top-5 accuracy, respectively). We
further design a hardware processor to address the hardware inefficiency caused
by floating-point multiplier. Compared with a state-of-the-art accelerator,
Phoenix is 3.32x and 7.45x better in performance with the same core area for
AlexNet and VGG16, respectively.
