  Deep learning models have introduced various intelligent applications to edge
devices, such as image classification, speech recognition, and augmented
reality. There is an increasing need of training such models on the devices in
order to deliver personalized, responsive, and private learning. To address
this need, this paper presents a new solution for deploying and training
state-of-the-art models on the resource-constrained devices. First, the paper
proposes a novel filter-pruning-based model compression method to create
lightweight trainable models from large models trained in the cloud, without
much loss of accuracy. Second, it proposes a novel knowledge transfer method to
enable the on-device model to update incrementally in real time or near real
time using incremental learning on new data and enable the on-device model to
learn the unseen categories with the help of the in-cloud model in an
unsupervised fashion. The results show that 1) our model compression method can
remove up to 99.36% parameters of WRN-28-10, while preserving a Top-1 accuracy
of over 90% on CIFAR-10; 2) our knowledge transfer method enables the
compressed models to achieve more than 90% accuracy on CIFAR-10 and retain good
accuracy on old categories; 3) it allows the compressed models to converge
within real time (three to six minutes) on the edge for incremental learning
tasks; 4) it enables the model to classify unseen categories of data (78.92%
Top-1 accuracy) that it is never trained with.
