  Emerging Artificial Intelligence of Things (AIoT) applications desire online
prediction using deep neural network (DNN) models on mobile devices. However,
due to the movement of devices, unfamiliar test samples constantly appear,
significantly affecting the prediction accuracy of a pre-trained DNN. In
addition, unstable network connection calls for local model inference. In this
paper, we propose a light-weight scheme, called Anole, to cope with the local
DNN model inference on mobile devices. The core idea of Anole is to first
establish an army of compact DNN models, and then adaptively select the model
fitting the current test sample best for online inference. The key is to
automatically identify model-friendly scenes for training scene-specific DNN
models. To this end, we design a weakly-supervised scene representation
learning algorithm by combining both human heuristics and feature similarity in
separating scenes. Moreover, we further train a model classifier to predict the
best-fit scene-specific DNN model for each test sample. We implement Anole on
different types of mobile devices and conduct extensive trace-driven and
real-world experiments based on unmanned aerial vehicles (UAVs). The results
demonstrate that Anole outwits the method of using a versatile large DNN in
terms of prediction accuracy (4.5% higher), response time (33.1% faster) and
power consumption (45.1% lower).
