  Despite living in a multi-sensory world, most AI models are limited to
textual and visual understanding of human motion and behavior. In fact, full
situational awareness of human motion could best be understood through a
combination of sensors. In this survey we investigate how knowledge can be
transferred and utilized amongst modalities for Human Activity/Action
Recognition (HAR), i.e. cross-modality transfer learning. We motivate the
importance and potential of IMU data and its applicability in cross-modality
learning as well as the importance of studying the HAR problem. We categorize
HAR related tasks by time and abstractness and then compare various types of
multimodal HAR datasets. We also distinguish and expound on many related but
inconsistently used terms in the literature, such as transfer learning, domain
adaptation, representation learning, sensor fusion, and multimodal learning,
and describe how cross-modal learning fits with all these concepts. We then
review the literature in IMU-based cross-modal transfer for HAR. The two main
approaches for cross-modal transfer are instance-based transfer, where
instances of one modality are mapped to another (e.g. knowledge is transferred
in the input space), or feature-based transfer, where the model relates the
modalities in an intermediate latent space (e.g. knowledge is transferred in
the feature space). Finally, we discuss future research directions and
applications in cross-modal HAR.
