  Despite living in a multi-sensory world, most AI models are limited to
textual and visual interpretations of human motion and behavior. Inertial
measurement units (IMUs) provide a salient signal to understand human motion;
however, they are challenging to use due to their uninterpretability and
scarcity of their data. We investigate a method to transfer knowledge between
visual and inertial modalities using the structure of an informative joint
representation space designed for human action recognition (HAR). We apply the
resulting Fusion and Cross-modal Transfer (FACT) method to a novel setup, where
the model does not have access to labeled IMU data during training and is able
to perform HAR with only IMU data during testing. Extensive experiments on a
wide range of RGB-IMU datasets demonstrate that FACT significantly outperforms
existing methods in zero-shot cross-modal transfer.
