  This paper introduces a manually annotated video dataset of unusual actions,
namely RareAct, including actions such as "blend phone", "cut keyboard" and
"microwave shoes". RareAct aims at evaluating the zero-shot and few-shot
compositionality of action recognition models for unlikely compositions of
common action verbs and object nouns. It contains 122 different actions which
were obtained by combining verbs and nouns rarely co-occurring together in the
large-scale textual corpus from HowTo100M, but that frequently appear
separately. We provide benchmarks using a state-of-the-art HowTo100M pretrained
video and text model and show that zero-shot and few-shot compositionality of
actions remains a challenging and unsolved task.
