  Zero-shot action recognition is the task of classifying action categories
that are not available in the training set. In this setting, the standard
evaluation protocol is to use existing action recognition datasets(e.g. UCF101)
and randomly split the classes into seen and unseen. However, most recent work
builds on representations pre-trained on the Kinetics dataset, where classes
largely overlap with classes in the zero-shot evaluation datasets. As a result,
classes which are supposed to be unseen, are present during supervised
pre-training, invalidating the condition of the zero-shot setting. A similar
concern was previously noted several years ago for image based zero-shot
recognition but has not been considered by the zero-shot action recognition
community. In this paper, we propose a new split for true zero-shot action
recognition with no overlap between unseen test classes and training or
pre-training classes. We benchmark several recent approaches on the proposed
True Zero-Shot(TruZe) Split for UCF101 and HMDB51, with zero-shot and
generalized zero-shot evaluation. In our extensive analysis, we find that our
TruZesplits are significantly harder than comparable random splits as nothing
is leaking from pre-training, i.e. unseen performance is consistently lower,up
to 8.9% for zero-shot action recognition. In an additional evaluation we also
find that similar issues exist in the splits used in few-shot action
recognition, here we see differences of up to 17.1%. We publish oursplits1and
hope that our benchmark analysis will change how the field is evaluating zero-
and few-shot action recognition moving forward.
