  Deep learning has achieved great success in recognizing video actions, but
the collection and annotation of training data are still quite laborious, which
mainly lies in two aspects: (1) the amount of required annotated data is large;
(2) temporally annotating the location of each action is time-consuming. Works
such as few-shot learning or untrimmed video recognition have been proposed to
handle either one aspect or the other. However, very few existing works can
handle both issues simultaneously. In this paper, we target a new problem,
Annotation-Efficient Video Recognition, to reduce the requirement of
annotations for both large amount of samples and the action location. Such
problem is challenging due to two aspects: (1) the untrimmed videos only have
weak supervision; (2) video segments not relevant to current actions of
interests (background, BG) could contain actions of interests (foreground, FG)
in novel classes, which is a widely existing phenomenon but has rarely been
studied in few-shot untrimmed video recognition. To achieve this goal, by
analyzing the property of BG, we categorize BG into informative BG (IBG) and
non-informative BG (NBG), and we propose (1) an open-set detection based method
to find the NBG and FG, (2) a contrastive learning method to learn IBG and
distinguish NBG in a self-supervised way, and (3) a self-weighting mechanism
for the better distinguishing of IBG and FG. Extensive experiments on
ActivityNet v1.2 and ActivityNet v1.3 verify the rationale and effectiveness of
the proposed methods.
