  Few-shot action recognition aims at quickly adapting a pre-trained model to
the novel data with a distribution shift using only a limited number of
samples. Key challenges include how to identify and leverage the transferable
knowledge learned by the pre-trained model. We therefore propose CDTD, or
Causal Domain-Invariant Temporal Dynamics for knowledge transfer. To identify
the temporally invariant and variant representations, we employ the causal
representation learning methods for unsupervised pertaining, and then tune the
classifier with supervisions in next stage. Specifically, we assume the domain
information can be well estimated and the pre-trained image decoder and
transition models can be well transferred. During adaptation, we fix the
transferable temporal dynamics and update the image encoder and domain
estimator. The efficacy of our approach is revealed by the superior accuracy of
CDTD over leading alternatives across standard few-shot action recognition
datasets.
