  Enabling efficient and accurate deep neural network (DNN) inference on
microcontrollers is non-trivial due to the constrained on-chip resources.
Current methodologies primarily focus on compressing larger models yet at the
expense of model accuracy. In this paper, we rethink the problem from the
inverse perspective by constructing small/weak models directly and improving
their accuracy. Thus, we introduce DiTMoS, a novel DNN training and inference
framework with a selector-classifiers architecture, where the selector routes
each input sample to the appropriate classifier for classification. DiTMoS is
grounded on a key insight: a composition of weak models can exhibit high
diversity and the union of them can significantly boost the accuracy upper
bound. To approach the upper bound, DiTMoS introduces three strategies
including diverse training data splitting to increase the classifiers'
diversity, adversarial selector-classifiers training to ensure synergistic
interactions thereby maximizing their complementarity, and heterogeneous
feature aggregation to improve the capacity of classifiers. We further propose
a network slicing technique to alleviate the extra memory overhead incurred by
feature aggregation. We deploy DiTMoS on the Neucleo STM32F767ZI board and
evaluate it based on three time-series datasets for human activity recognition,
keywords spotting, and emotion recognition, respectively. The experiment
results manifest that: (a) DiTMoS achieves up to 13.4% accuracy improvement
compared to the best baseline; (b) network slicing almost completely eliminates
the memory overhead incurred by feature aggregation with a marginal increase of
latency.
