  Existing vision-based action recognition is susceptible to occlusion and
appearance variations, while wearable sensors can alleviate these challenges by
capturing human motion with one-dimensional time-series signal. For the same
action, the knowledge learned from vision sensors and wearable sensors, may be
related and complementary. However, there exists significantly large modality
difference between action data captured by wearable-sensor and vision-sensor in
data dimension, data distribution and inherent information content. In this
paper, we propose a novel framework, named Semantics-aware Adaptive Knowledge
Distillation Networks (SAKDN), to enhance action recognition in vision-sensor
modality (videos) by adaptively transferring and distilling the knowledge from
multiple wearable sensors. The SAKDN uses multiple wearable-sensors as teacher
modalities and uses RGB videos as student modality. To preserve local temporal
relationship and facilitate employing visual deep learning model, we transform
one-dimensional time-series signals of wearable sensors to two-dimensional
images by designing a gramian angular field based virtual image generation
model. Then, we build a novel Similarity-Preserving Adaptive Multi-modal Fusion
Module to adaptively fuse intermediate representation knowledge from different
teacher networks. Finally, to fully exploit and transfer the knowledge of
multiple well-trained teacher networks to the student network, we propose a
novel Graph-guided Semantically Discriminative Mapping loss, which utilizes
graph-guided ablation analysis to produce a good visual explanation
highlighting the important regions across modalities and concurrently
preserving the interrelations of original data. Experimental results on
Berkeley-MHAD, UTD-MHAD and MMAct datasets well demonstrate the effectiveness
of our proposed SAKDN.
