  Recent advancements in large-scale pre-training of visual-language models on
paired image-text data have demonstrated impressive generalization capabilities
for zero-shot tasks. Building on this success, efforts have been made to adapt
these image-based visual-language models, such as CLIP, for videos extending
their zero-shot capabilities to the video domain. While these adaptations have
shown promising results, they come at a significant computational cost and
struggle with effectively modeling the crucial temporal aspects inherent to the
video domain. In this study, we present EZ-CLIP, a simple and efficient
adaptation of CLIP that addresses these challenges. EZ-CLIP leverages temporal
visual prompting for seamless temporal adaptation, requiring no fundamental
alterations to the core CLIP architecture while preserving its remarkable
generalization abilities. Moreover, we introduce a novel learning objective
that guides the temporal visual prompts to focus on capturing motion, thereby
enhancing its learning capabilities from video data. We conducted extensive
experiments on five different benchmark datasets, thoroughly evaluating EZ-CLIP
for zero-shot learning and base-to-novel video action recognition, and also
demonstrating its potential for few-shot generalization.Impressively, with a
mere 5.2 million learnable parameters (as opposed to the 71.1 million in the
prior best model), EZ-CLIP can be efficiently trained on a single GPU,
outperforming existing approaches in several evaluations.
