  In this work, we address the problem how a network for action recognition
that has been trained on a modality like RGB videos can be adapted to recognize
actions for another modality like sequences of 3D human poses. To this end, we
extract the knowledge of the trained teacher network for the source modality
and transfer it to a small ensemble of student networks for the target
modality. For the cross-modal knowledge distillation, we do not require any
annotated data. Instead we use pairs of sequences of both modalities as
supervision, which are straightforward to acquire. In contrast to previous
works for knowledge distillation that use a KL-loss, we show that the
cross-entropy loss together with mutual learning of a small ensemble of student
networks performs better. In fact, the proposed approach for cross-modal
knowledge distillation nearly achieves the accuracy of a student network
trained with full supervision.
