  This work introduces lightweight extensions to the RISC-V ISA to boost the
efficiency of heavily Quantized Neural Network (QNN) inference on
microcontroller-class cores. By extending the ISA with nibble (4-bit) and crumb
(2-bit) SIMD instructions, we are able to show near-linear speedup with respect
to higher precision integer computation on the key kernels for QNN computation.
Also, we propose a custom execution paradigm for SIMD sum-of-dot-product
operations, which consists of fusing a dot product with a load operation, with
an up to 1.64x peak MAC/cycle improvement compared to a standard execution
scenario. To further push the efficiency, we integrate the RISC-V extended core
in a parallel cluster of 8 processors, with near-linear improvement with
respect to a single core architecture. To evaluate the proposed extensions, we
fully implement the cluster of processors in GF22FDX technology. QNN
convolution kernels on a parallel cluster implementing the proposed extension
run 6 x and 8 x faster when considering 4- and 2-bit data operands,
respectively, compared to a baseline processing cluster only supporting 8-bit
SIMD instructions. With a peak of 2.22 TOPs/s/W, the proposed solution achieves
efficiency levels comparable with dedicated DNN inference accelerators, and up
to three orders of magnitude better than state-of-the-art ARM Cortex-M based
microcontroller systems such as the low-end STM32L4 MCU and the high-end
STM32H7 MCU.
