  Localizing people and recognizing their actions from videos is a challenging
task towards high-level video understanding. Existing methods are mostly
two-stage based, with one stage for person bounding box generation and the
other stage for action recognition. However, such two-stage methods are
generally with low efficiency. We observe that directly unifying detection and
action recognition normally suffers from (i) inferior learning due to different
desired properties of context representation for detection and action
recognition; (ii) optimization difficulty with insufficient training data. In
this work, we present a decoupled one-stage network dubbed DOAD, to mitigate
above issues and improve the efficiency for spatio-temporal action detection.
To achieve it, we decouple detection and action recognition into two branches.
Specifically, one branch focuses on detection representation for actor
detection, and the other one for action recognition. For the action branch, we
design a transformer-based module (TransPC) to model pairwise relationships
between people and context. Different from commonly used vector-based dot
product in self-attention, it is built upon a novel matrix-based key and value
for Hadamard attention to model person-context information. It not only
exploits relationships between person pairs but also takes into account context
and relative position information. The results on AVA and UCF101-24 datasets
show that our method is competitive with two-stage state-of-the-art methods
with significant efficiency improvement.
