  There is a growing interest in learning a model which could recognize novel
classes with only a few labeled examples. In this paper, we propose Temporal
Alignment Module (TAM), a novel few-shot learning framework that can learn to
classify a previous unseen video. While most previous works neglect long-term
temporal ordering information, our proposed model explicitly leverages the
temporal ordering information in video data through temporal alignment. This
leads to strong data-efficiency for few-shot learning. In concrete, TAM
calculates the distance value of query video with respect to novel class
proxies by averaging the per frame distances along its alignment path. We
introduce continuous relaxation to TAM so the model can be learned in an
end-to-end fashion to directly optimize the few-shot learning objective. We
evaluate TAM on two challenging real-world datasets, Kinetics and
Something-Something-V2, and show that our model leads to significant
improvement of few-shot video classification over a wide range of competitive
baselines.
