  Human activity recognition (HAR) based on multi-modal approach has been
recently shown to improve the accuracy performance of HAR. However, restricted
computational resources associated with wearable devices, i.e., smartwatch,
failed to directly support such advanced methods. To tackle this issue, this
study introduces an end-to-end Vision-to-Sensor Knowledge Distillation (VSKD)
framework. In this VSKD framework, only time-series data, i.e., accelerometer
data, is needed from wearable devices during the testing phase. Therefore, this
framework will not only reduce the computational demands on edge devices, but
also produce a learning model that closely matches the performance of the
computational expensive multi-modal approach. In order to retain the local
temporal relationship and facilitate visual deep learning models, we first
convert time-series data to two-dimensional images by applying the Gramian
Angular Field ( GAF) based encoding method. We adopted ResNet18 and multi-scale
TRN with BN-Inception as teacher and student network in this study,
respectively. A novel loss function, named Distance and Angle-wised Semantic
Knowledge loss (DASK), is proposed to mitigate the modality variations between
the vision and the sensor domain. Extensive experimental results on UTD-MHAD,
MMAct, and Berkeley-MHAD datasets demonstrate the effectiveness and
competitiveness of the proposed VSKD model which can deployed on wearable
sensors.
