  Wearable sensor-based Human Action Recognition (HAR) has achieved remarkable
success recently. However, the accuracy performance of wearable sensor-based
HAR is still far behind the ones from the visual modalities-based system (i.e.,
RGB video, skeleton, and depth). Diverse input modalities can provide
complementary cues and thus improve the accuracy performance of HAR, but how to
take advantage of multi-modal data on wearable sensor-based HAR has rarely been
explored. Currently, wearable devices, i.e., smartwatches, can only capture
limited kinds of non-visual modality data. This hinders the multi-modal HAR
association as it is unable to simultaneously use both visual and non-visual
modality data. Another major challenge lies in how to efficiently utilize
multimodal data on wearable devices with their limited computation resources.
In this work, we propose a novel Progressive Skeleton-to-sensor Knowledge
Distillation (PSKD) model which utilizes only time-series data, i.e.,
accelerometer data, from a smartwatch for solving the wearable sensor-based HAR
problem. Specifically, we construct multiple teacher models using data from
both teacher (human skeleton sequence) and student (time-series accelerometer
data) modalities. In addition, we propose an effective progressive learning
scheme to eliminate the performance gap between teacher and student models. We
also designed a novel loss function called Adaptive-Confidence Semantic (ACS),
to allow the student model to adaptively select either one of the teacher
models or the ground-truth label it needs to mimic. To demonstrate the
effectiveness of our proposed PSKD method, we conduct extensive experiments on
Berkeley-MHAD, UTD-MHAD, and MMAct datasets. The results confirm that the
proposed PSKD method has competitive performance compared to the previous mono
sensor-based HAR methods.
