  Spatio-temporal action detection (STAD) is an important fine-grained video
understanding task. Current methods require box and label supervision for all
action classes in advance. However, in real-world applications, it is very
likely to come across new action classes not seen in training because the
action category space is large and hard to enumerate. Also, the cost of data
annotation and model training for new classes is extremely high for traditional
methods, as we need to perform detailed box annotations and re-train the whole
network from scratch. In this paper, we propose a new challenging setting by
performing open-vocabulary STAD to better mimic the situation of action
detection in an open world. Open-vocabulary spatio-temporal action detection
(OV-STAD) requires training a model on a limited set of base classes with box
and label supervision, which is expected to yield good generalization
performance on novel action classes. For OV-STAD, we build two benchmarks based
on the existing STAD datasets and propose a simple but effective method based
on pretrained video-language models (VLM). To better adapt the holistic VLM for
the fine-grained action detection task, we carefully fine-tune it on the
localized video region-text pairs. This customized fine-tuning endows the VLM
with better motion understanding, thus contributing to a more accurate
alignment between video regions and texts. Local region feature and global
video feature fusion before alignment is adopted to further improve the action
detection performance by providing global context. Our method achieves a
promising performance on novel classes.
