  Compressed video action recognition classifies video samples by leveraging
the different modalities in compressed videos, namely motion vectors,
residuals, and intra-frames. For this purpose, three neural networks are
deployed, each dedicated to processing one modality. Our observations indicate
that the network processing intra-frames tend to converge to a flatter minimum
than the network processing residuals, which in turn converges to a flatter
minimum than the motion vector network. This hierarchy in convergence motivates
our strategy for knowledge transfer among modalities to achieve flatter minima,
which are generally associated with better generalization. With this insight,
we propose Progressive Knowledge Distillation (PKD), a technique that
incrementally transfers knowledge across the modalities. This method involves
attaching early exits (Internal Classifiers - ICs) to the three networks. PKD
distills knowledge starting from the motion vector network, followed by the
residual, and finally, the intra-frame network, sequentially improving IC
accuracy. Further, we propose the Weighted Inference with Scaled Ensemble
(WISE), which combines outputs from the ICs using learned weights, boosting
accuracy during inference. Our experiments demonstrate the effectiveness of
training the ICs with PKD compared to standard cross-entropy-based training,
showing IC accuracy improvements of up to 5.87% and 11.42% on the UCF-101 and
HMDB-51 datasets, respectively. Additionally, WISE improves accuracy by up to
4.28% and 9.30% on UCF-101 and HMDB-51, respectively.
