  The growing number of low-power smart devices in the Internet of Things is
coupled with the concept of "Edge Computing", that is moving some of the
intelligence, especially machine learning, towards the edge of the network.
Enabling machine learning algorithms to run on resource-constrained hardware,
typically on low-power smart devices, is challenging in terms of hardware
(optimized and energy-efficient integrated circuits), algorithmic and firmware
implementations. This paper presents FANN-on-MCU, an open-source toolkit built
upon the Fast Artificial Neural Network (FANN) library to run lightweight and
energy-efficient neural networks on microcontrollers based on both the ARM
Cortex-M series and the novel RISC-V-based Parallel Ultra-Low-Power (PULP)
platform. The toolkit takes multi-layer perceptrons trained with FANN and
generates code targeted at execution on low-power microcontrollers either with
a floating-point unit (i.e., ARM Cortex-M4F and M7F) or without (i.e., ARM
Cortex M0-M3 or PULP-based processors). This paper also provides an
architectural performance evaluation of neural networks on the most popular ARM
Cortex-M family and the parallel RISC-V processor called Mr. Wolf. The
evaluation includes experimental results for three different applications using
a self-sustainable wearable multi-sensor bracelet. Experimental results show a
measured latency in the order of only a few microseconds and a power
consumption of few milliwatts while keeping the memory requirements below the
limitations of the targeted microcontrollers. In particular, the parallel
implementation on the octa-core RISC-V platform reaches a speedup of 22x and a
69% reduction in energy consumption with respect to a single-core
implementation on Cortex-M4 for continuous real-time classification.
