  Recent advancements in quantization and mixed-precision approaches offers
substantial opportunities to improve the speed and energy efficiency of Neural
Networks (NN). Research has shown that individual parameters with varying low
precision, can attain accuracies comparable to full-precision counterparts.
However, modern embedded microprocessors provide very limited support for
mixed-precision NNs regarding both Instruction Set Architecture (ISA)
extensions and their hardware design for efficient execution of mixed-precision
operations, i.e., introducing several performance bottlenecks due to numerous
instructions for data packing and unpacking, arithmetic unit under-utilizations
etc. In this work, we bring together, for the first time, ISA extensions
tailored to mixed-precision hardware optimizations, targeting energy-efficient
DNN inference on leading RISC-V CPU architectures. To this end, we introduce a
hardware-software co-design framework that enables cooperative hardware design,
mixed-precision quantization, ISA extensions and inference in cycle-accurate
emulations. At hardware level, we firstly expand the ALU unit within our
proof-of-concept micro-architecture to support configurable fine grained
mixed-precision arithmetic operations. Subsequently, we implement multi-pumping
to minimize execution latency, with an additional soft SIMD optimization
applied for 2-bit operations. At the ISA level, three distinct MAC instructions
are encoded extending the RISC-V ISA, and exposed up to the compiler level,
each corresponding to a different mixed-precision operational mode. Our
extensive experimental evaluation over widely used DNNs and datasets, such as
CIFAR10 and ImageNet, demonstrates that our framework can achieve, on average,
15x energy reduction for less than 1% accuracy loss and outperforms the
ISA-agnostic state-of-the-art RISC-V cores.
