  Spatio-temporal action detection in videos requires localizing the action
both spatially and temporally in the form of an "action tube". Nowadays, most
spatio-temporal action detection datasets (e.g. UCF101-24, AVA, DALY) are
annotated with action tubes that contain a single person performing the action,
thus the predominant action detection models simply employ a person detection
and tracking pipeline for localization. However, when the action is defined as
an interaction between multiple objects, such methods may fail since each
bounding box in the action tube contains multiple objects instead of one
person. In this paper, we study the spatio-temporal action detection problem
with multi-object interaction. We introduce a new dataset that is annotated
with action tubes containing multi-object interactions. Moreover, we propose an
end-to-end spatio-temporal action detection model that performs both spatial
and temporal regression simultaneously. Our spatial regression may enclose
multiple objects participating in the action. During test time, we simply
connect the regressed bounding boxes within the predicted temporal duration
using a simple heuristic. We report the baseline results of our proposed model
on this new dataset, and also show competitive results on the standard
benchmark UCF101-24 using only RGB input.
