  Recently low-precision deep learning accelerators (DLAs) have become popular
due to their advantages in chip area and energy consumption, yet the
low-precision quantized models on these DLAs bring in severe accuracy
degradation. One way to achieve both high accuracy and efficient inference is
to deploy high-precision neural networks on low-precision DLAs, which is rarely
studied. In this paper, we propose the PArallel Low-precision Quantization
(PalQuant) method that approximates high-precision computations via learning
parallel low-precision representations from scratch. In addition, we present a
novel cyclic shuffle module to boost the cross-group information communication
between parallel low-precision groups. Extensive experiments demonstrate that
PalQuant has superior performance to state-of-the-art quantization methods in
both accuracy and inference speed, e.g., for ResNet-18 network quantization,
PalQuant can obtain 0.52\% higher accuracy and 1.78$\times$ speedup
simultaneously over their 4-bit counter-part on a state-of-the-art 2-bit
accelerator. Code is available at \url{https://github.com/huqinghao/PalQuant}.
