  We explore the use of knowledge distillation (KD) for learning compact and
accurate models that enable classification of animal behavior from
accelerometry data on wearable devices. To this end, we take a deep and complex
convolutional neural network, known as residual neural network (ResNet), as the
teacher model. ResNet is specifically designed for multivariate time-series
classification. We use ResNet to distill the knowledge of animal behavior
classification datasets into soft labels, which consist of the predicted
pseudo-probabilities of every class for each datapoint. We then use the soft
labels to train our significantly less complex student models, which are based
on the gated recurrent unit (GRU) and multilayer perceptron (MLP). The
evaluation results using two real-world animal behavior classification datasets
show that the classification accuracy of the student GRU-MLP models improves
appreciably through KD, approaching that of the teacher ResNet model. To
further reduce the computational and memory requirements of performing
inference using the student models trained via KD, we utilize dynamic
fixed-point quantization (DQ) through an appropriate modification of the
computational graph of the considered models. We implement both unquantized and
quantized versions of the developed KD-based models on the embedded systems of
our purpose-built collar and ear tag devices to classify animal behavior in
situ and in real time. Our evaluations corroborate the effectiveness of KD and
DQ in improving the accuracy and efficiency of in-situ animal behavior
classification.
