  We present PULP-NN, an optimized computing library for a parallel
ultra-low-power tightly coupled cluster of RISC-V processors. The key
innovation in PULP-NN is a set of kernels for Quantized Neural Network (QNN)
inference, targeting byte and sub-byte data types, down to INT-1, tuned for the
recent trend toward aggressive quantization in deep neural network inference.
The proposed library exploits both the digital signal processing (DSP)
extensions available in the PULP RISC-V processors and the cluster's
parallelism, achieving up to 15.5 MACs/cycle on INT-8 and improving performance
by up to 63x with respect to a sequential implementation on a single RISC-V
core implementing the baseline RV32IMC ISA. Using PULP-NN, a CIFAR-10 network
on an octa-core cluster runs in 30x and 19.6x less clock cycles than the
current state-of-the-art ARM CMSIS-NN library, running on STM32L4 and STM32H7
MCUs, respectively. The proposed library, when running on GAP-8 processor,
outperforms by 36.8x and by 7.45x the execution on energy efficient MCUs such
as STM32L4 and high-end MCUs such as STM32H7 respectively, when operating at
the maximum frequency. The energy efficiency on GAP-8 is 14.1x higher than
STM32L4 and 39.5x higher than STM32H7, at the maximum efficiency operating
point.
