  Spatiotemporal action localization requires the incorporation of two sources
of information into the designed architecture: (1) temporal information from
the previous frames and (2) spatial information from the key frame. Current
state-of-the-art approaches usually extract these information with separate
networks and use an extra mechanism for fusion to get detections. In this work,
we present YOWO, a unified CNN architecture for real-time spatiotemporal action
localization in video streams. YOWO is a single-stage architecture with two
branches to extract temporal and spatial information concurrently and predict
bounding boxes and action probabilities directly from video clips in one
evaluation. Since the whole architecture is unified, it can be optimized
end-to-end. The YOWO architecture is fast providing 34 frames-per-second on
16-frames input clips and 62 frames-per-second on 8-frames input clips, which
is currently the fastest state-of-the-art architecture on spatiotemporal action
localization task. Remarkably, YOWO outperforms the previous state-of-the art
results on J-HMDB-21 and UCF101-24 with an impressive improvement of ~3% and
~12%, respectively. Moreover, YOWO is the first and only single-stage
architecture that provides competitive results on AVA dataset. We make our code
and pretrained models publicly available.
