  Executing machine learning workloads locally on resource constrained
microcontrollers (MCUs) promises to drastically expand the application space of
IoT. However, so-called TinyML presents severe technical challenges, as deep
neural network inference demands a large compute and memory budget. To address
this challenge, neural architecture search (NAS) promises to help design
accurate ML models that meet the tight MCU memory, latency and energy
constraints. A key component of NAS algorithms is their latency/energy model,
i.e., the mapping from a given neural network architecture to its inference
latency/energy on an MCU. In this paper, we observe an intriguing property of
NAS search spaces for MCU model design: on average, model latency varies
linearly with model operation (op) count under a uniform prior over models in
the search space. Exploiting this insight, we employ differentiable NAS (DNAS)
to search for models with low memory usage and low op count, where op count is
treated as a viable proxy to latency. Experimental results validate our
methodology, yielding our MicroNet models, which we deploy on MCUs using
Tensorflow Lite Micro, a standard open-source NN inference runtime widely used
in the TinyML community. MicroNets demonstrate state-of-the-art results for all
three TinyMLperf industry-standard benchmark tasks: visual wake words, audio
keyword spotting, and anomaly detection. Models and training scripts can be
found at github.com/ARM-software/ML-zoo.
