  Designing a real-time framework for the spatio-temporal action detection task
is still a challenge. In this paper, we propose a novel real-time action
detection framework, YOWOv2. In this new framework, YOWOv2 takes advantage of
both the 3D backbone and 2D backbone for accurate action detection. A
multi-level detection pipeline is designed to detect action instances of
different scales. To achieve this goal, we carefully build a simple and
efficient 2D backbone with a feature pyramid network to extract different
levels of classification features and regression features. For the 3D backbone,
we adopt the existing efficient 3D CNN to save development time. By combining
3D backbones and 2D backbones of different sizes, we design a YOWOv2 family
including YOWOv2-Tiny, YOWOv2-Medium, and YOWOv2-Large. We also introduce the
popular dynamic label assignment strategy and anchor-free mechanism to make the
YOWOv2 consistent with the advanced model architecture design. With our
improvement, YOWOv2 is significantly superior to YOWO, and can still keep
real-time detection. Without any bells and whistles, YOWOv2 achieves 87.0 %
frame mAP and 52.8 % video mAP with over 20 FPS on the UCF101-24. On the AVA,
YOWOv2 achieves 21.7 % frame mAP with over 20 FPS. Our code is available on
https://github.com/yjh0410/YOWOv2.
