  Current methods for few-shot action recognition mainly fall into the metric
learning framework following ProtoNet, which demonstrates the importance of
prototypes. Although they achieve relatively good performance, the effect of
multimodal information is ignored, e.g. label texts. In this work, we propose a
novel MultimOdal PRototype-ENhanced Network (MORN), which uses the semantic
information of label texts as multimodal information to enhance prototypes. A
CLIP visual encoder and a frozen CLIP text encoder are introduced to obtain
features with good multimodal initialization. Then in the visual flow, visual
prototypes are computed by a visual prototype-computed module. In the text
flow, a semantic-enhanced (SE) module and an inflating operation are used to
obtain text prototypes. The final multimodal prototypes are then computed by a
multimodal prototype-enhanced (MPE) module. Besides, we define a PRototype
SImilarity DiffErence (PRIDE) to evaluate the quality of prototypes, which is
used to verify our improvement on the prototype level and effectiveness of
MORN. We conduct extensive experiments on four popular few-shot action
recognition datasets: HMDB51, UCF101, Kinetics and SSv2, and MORN achieves
state-of-the-art results. When plugging PRIDE into the training stage, the
performance can be further improved.
