  Spatio-temporal action detection is an important and challenging problem in
video understanding. The existing action detection benchmarks are limited in
aspects of small numbers of instances in a trimmed video or low-level atomic
actions. This paper aims to present a new multi-person dataset of
spatio-temporal localized sports actions, coined as MultiSports. We first
analyze the important ingredients of constructing a realistic and challenging
dataset for spatio-temporal action detection by proposing three criteria: (1)
multi-person scenes and motion dependent identification, (2) with well-defined
boundaries, (3) relatively fine-grained classes of high complexity. Based on
these guide-lines, we build the dataset of MultiSports v1.0 by selecting 4
sports classes, collecting 3200 video clips, and annotating 37701 action
instances with 902k bounding boxes. Our datasets are characterized with
important properties of high diversity, dense annotation, and high quality. Our
Multi-Sports, with its realistic setting and detailed annotations, exposes the
intrinsic challenges of spatio-temporal action detection. To benchmark this, we
adapt several baseline methods to our dataset and give an in-depth analysis on
the action detection results in our dataset. We hope our MultiSports can serve
as a standard benchmark for spatio-temporal action detection in the future. Our
dataset website is at https://deeperaction.github.io/multisports/.
