  Crossmodal knowledge distillation (KD) extends traditional knowledge
distillation to the area of multimodal learning and demonstrates great success
in various applications. To achieve knowledge transfer across modalities, a
pretrained network from one modality is adopted as the teacher to provide
supervision signals to a student network learning from another modality. In
contrast to the empirical success reported in prior works, the working
mechanism of crossmodal KD remains a mystery. In this paper, we present a
thorough understanding of crossmodal KD. We begin with two case studies and
demonstrate that KD is not a universal cure in crossmodal knowledge transfer.
We then present the modality Venn diagram to understand modality relationships
and the modality focusing hypothesis revealing the decisive factor in the
efficacy of crossmodal KD. Experimental results on 6 multimodal datasets help
justify our hypothesis, diagnose failure cases, and point directions to improve
crossmodal knowledge transfer in the future.
